Source-linked AI summary
Efficient Optical flow and Stereo Vision for Velocity Estimation and Obstacle Avoidance on an Autonomous Pocket Drone
Kimberly McGuire, Guido de Croon, Christophe De Wagter, Karl Tuyls, Hilbert Kappen
TL;DR
Autonomous indoor flight on pocket drones is constrained by limited onboard energy, sensing, and processing resources. The paper presents Edge-FS, which combines efficient edge-based optical flow with stereo depth to estimate velocity and distance on a lightweight embedded stereo camera. On a 40 g pocket drone, these measurements support autonomous navigation and obstacle avoidance using onboard sensing.
Problem
Limited energy, sensing, and processing capabilities make conventional optical-flow and stereo-vision methods difficult to run onboard pocket drones for autonomous navigation.
Method
Edge-FS combines EdgeFlow optical flow with EdgeStereo depth estimation and scales local flow using stereo-based distances to retrieve velocity.
Results
The method ran on a lightweight stereo camera carried by a 40 g pocket drone and supported autonomous navigation and obstacle avoidance in a 4 x 4 meter area.
Takeaways & Limitations
Velocity and depth from one small forward-facing stereo vision system can support fully autonomous flight and obstacle avoidance on a pocket drone.
Takeaways & Limitations
Forward-velocity estimation is more noise-sensitive than sideways estimation, especially during large lateral motion, while stereo vision also has difficulty at larger distances.
Abstract
from arXiv · showhide
Miniature Micro Aerial Vehicles (MAV) are very suitable for flying in indoor environments, but autonomous navigation is challenging due to their strict hardware limitations. This paper presents a highly efficient computer vision algorithm called Edge-FS for the determination of velocity and depth. It runs at 20 Hz on a 4 g stereo camera with an embedded STM32F4 microprocessor (168 MHz, 192 kB) and uses feature histograms to calculate optical flow and stereo disparity. The stereo-based distance estimates are used to scale the optical flow in order to retrieve the drone's velocity. The velocity and depth measurements are used for fully autonomous flight of a 40 g pocket drone only relying on on-board sensors. The method allows the MAV to control its velocity and avoid obstacles.
I. INTRODUCTION
Pocket drones offer energy-efficient visual sensing for indoor autonomy, but limited onboard computation makes conventional optical-flow and stereo methods impractical. The paper extends efficient edge-based vision to estimate velocity and depth for forward-looking navigation and autonomous obstacle avoidance.
- Pocket drones under 50 g suit narrow indoor spaces, where RGB cameras can observe obstacles, motion, and objects with low energy use.
- Strict energy, sensing, and processing limits make even efficient stereo-vision methods too computationally heavy for onboard pocket-drone processors.
- Standard optical-flow methods may not run in real time on pocket drones or may consume processing capacity needed for other autonomous-flight tasks.
- EdgeFlow previously ran at 30 Hz on a lightweight stereo camera but assumed downward viewing over a flat ground surface.
- Edge-FS extends EdgeFlow with EdgeStereo to estimate optical flow and stereo depth across a forward-facing image, scaling flow by distance for velocity estimation.
- The method provides velocity and distance estimates near the frame rate on a limited embedded processor and supports autonomous flight in unknown environments.
A. Related Work
Related work demonstrates lightweight optical-flow and stereo-based control, but prior systems often relied on external computation, multiple single-purpose sensors, or platforms larger than a pocket drone.
- 25 g quadcopter visual odometry was demonstrated with optical flow computed on an external computer, introducing potential control delays from communication bandwidth limits.
- A 45 g quadcopter used multiple one-dimensional flow sensors for optical-flow control and later a 278 g platform used eight sensors for hovering in cluttered environments.
- A 30 g helicopter computed translational optical flow onboard at 10 Hz using eight 64 x 64 image chips, while flight controls remained off-board.
- Stereo disparity has supported reactive obstacle avoidance without requiring the MAV to remain constantly in motion.
II. VELOCITY AND DEPTH FROM EDGES
The method requires velocity and depth for autonomous navigation, using edge-based optical flow and stereo depth to account for scene-dependent motion parallax rather than a single global height.
- Autonomous navigation on an unstable pocket drone requires both velocity and depth measurements.
A. From Camera to State
Forward-facing navigation requires estimating velocity and depth because non-planar scenes make a single global height insufficient for scaling optical flow. The method corrects rotational flow, obtains stereo depth, and fits scaled flow to recover forward and sideways velocity.
- From Camera to State: The coordinate system relates the camera’s body-fixed XYZ axes to aircraft east-north-up coordinates, while image width and height use u and v.This convention defines the axes used for the velocity equations.
- From Camera to State: Measured optical flow is separated into translational and rotational components before estimating horizontal velocity.The rotational component is approximated from yaw rotation measured by the on-board gyroscopes under small-angle assumptions.
- From Camera to State: Non-planar forward scenes require depth across the image because a global height cannot correctly scale optical flow around obstacles at different distances.Closer objects appear to move faster than farther objects, producing motion parallax.
- From Camera to State: EdgeFlow temporal disparities are scaled by EdgeStereo depth, then a linear fit extracts forward velocity from the slope and sideways velocity from the intercept.The fitting operates on depth-scaled translational optical flow across image columns.
- From Camera to State: Stereo disparity between the left and right images is converted into depth using the camera baseline and parameters.Depth is represented along image columns for subsequent optical-flow scaling.
B. Procedure for Edge-FS
Edge-FS reuses an efficient edge-distribution matching procedure for temporal optical flow and column-wise stereo disparity. Adaptive temporal matching and yaw compensation support the resulting depth-scaled flow computation.
- Procedure for Edge-FS: Sobel-filtered horizontal gradients are summed along image rows into edge distributions, which are compared using SAD block matching to find pixel displacement.The same matching principle underlies both EdgeFlow and EdgeStereo.
- Procedure for Edge-FS: For temporal image pairs, the matched displacement becomes pixel flow, while EdgeFlow adaptively selects the comparison horizon t−n from previous flow.Yaw-predicted flow shifts the start of block matching.
- Procedure for Edge-FS: The forward-facing version omits image-height flow for forward velocity because vx is subtracted from Edge-FS divergence.This differs from the earlier downward-looking configuration.
- Procedure for Edge-FS: EdgeStereo applies the same matching principle column-wise to obtain disparity, converts disparity to metric depth, and scales EdgeFlow for motion-parallax compensation.Column-wise processing supports velocity estimation with a forward-facing camera.
III. OFF-LINE VISION EXPERIMENTS
The off-line experiments evaluate Edge-FS velocity estimates against Färneback and ground truth across forward and sideways motion, while examining how depth affects accuracy.
- Edge-FS combines EdgeFlow velocity estimation with EdgeStereo scaling to estimate velocity from stereo-image data.The resulting method is evaluated on images captured while the camera moves toward obstacles at different distances.
- Edge-FS is compared with Färneback using 128 x 96 pixel images and matched window and search-range settings.Both methods estimate forward (x) and sideways (y) velocity against OptiTrack ground truth.
- All reported metrics indicate more accurate velocity estimates for Edge-FS than for the computationally more expensive Färneback method on this data set.The comparison uses Mean Squared Error, Variance, and Normalized Maximum Cross-Correlation Magnitude.
- Farther distances are measured less accurately because stereo disparities and translational optical flow become sub-pixel.The resulting distance–accuracy correlation is shown in the absolute velocity-error boxplot.
- 0.00134 seconds is the reported mean computation time for embedded EdgeFlow, compared with 0.00466 seconds for Färneback on the same stereo-image data set.
- Forward velocity is generally harder to estimate than sideways velocity, particularly under large lateral motion, because it is captured through optical-flow divergence.The experiments use larger SAD windows and filtering to reduce remaining noise during predominantly forward flight.
IV. EXPERIMENTS ON THE POCKET DRONE
The pocket-drone experiments use Edge-FS in a closed-loop flight to maintain velocity while autonomously navigating through a room and avoiding obstacles.
- The experiments progress from onboard velocity estimates to a closed-loop autonomous flight through a room with velocity maintenance and obstacle avoidance.
A. Hardware specifics
A 4 g front-facing stereo camera with an embedded STM32F4 is mounted on a 41.9 g pocket-drone assembly under tight imaging, memory, and processing constraints.
- The stereo camera uses two 1/6 inch sensors, a 6 cm baseline, a 57.5o x 44.5o FOV, and 128 x 96 pixel stereo images.
- The camera embeds an STM32F4 processor running at 168 MHz with 196 kB of memory, including a largest consecutive block of 128 kB.
- 30 Hz is the stereo camera’s maximum frame rate, with Edge-FS computation taking approximately 0.0175 seconds.
- The complete pocket-drone assembly, including stereo camera and battery, weighs exactly 41.9 g.
B. Velocity Estimate
The onboard flight evaluation tests whether Edge-FS remains useful under motion and vibration, then applies its measurements in a vision-only flight system.
- B. Velocity Estimate: The flight test asks whether Edge-FS retains velocity-estimation quality after installation on the stereo board and pocket drone, including motion and vibration effects.
- B. Velocity Estimate: During manually controlled flight before a textured screen, Edge-FS velocity estimates are compared with OptiTrack ground truth using NMXM, VAR, and MSE.
- B. Velocity Estimate: Forward velocity contains more noise peaks than sideways velocity, but Edge-FS matches ground truth adequately in both directions for the closed-loop flight.
- B. Velocity Estimate: The flight system uses vision-only velocity measurements with a median filter because nearby propeller vibrations make accelerometer readings too noisy.
C. Autonomous Obstacle Avoidance
The pocket drone uses Edge-FS velocity estimates and EdgeStereo depth measurements within a finite-state controller to navigate and avoid obstacles autonomously. Flights demonstrated sustained operation, but limited field of view and stereo range caused failures near walls.
- Obstacle-avoidance logic: EdgeStereo identifies the nearest object and generates a backward velocity reference when it is too close, preventing face-on wall collisions.The reference is sent to the velocity guidance controller, which sets an attitude set-point for stabilization.
- Obstacle-avoidance logic: The controller checks for obstacles within 1 meter, advances at 0.3 m/s when clear, then hovers and turns 60° after detecting an obstacle.This four-state finite-state machine uses active forward-velocity control during the 1-second hover before reassessing the path.
- Flight experiments: Three representative flights in a 4 × 4 meter room lasted over 90 seconds, with the longest lasting 122 seconds.The tests used varying textured surfaces, and additional flights were conducted in a real-world office environment.
- Flight experiments: The main failure occurred when the drone flew nearly parallel to a wall after turning, because the wall fell outside the camera’s limited field of view.Flights 2 and 3 exhibited this case; a safety pilot took over when the drone brushed a wall.
- Limitations: Far open space can degrade EdgeStereo depth and velocity quality because the stereo camera has a small baseline, causing drift near obstacles.Suggested remedies include merging the finite-state machine’s check and turn nodes or adding lightweight side-facing short-range sensors.
- Flight experiments: Despite needing additional obstacle-avoidance work, the experiments showed that Edge-FS stabilized the drone using velocity estimates from its forward camera alone.The obstacle-avoidance experiments therefore support navigation use of the velocity estimates, while not eliminating the identified failure modes.
V. CONCLUSION
The paper presents Edge-FS, an efficient optical-flow and stereo algorithm for forward-facing sensing on a lightweight pocket drone. Experiments showed autonomous navigation and obstacle avoidance in a 4 × 4 meter area, supporting velocity-based stabilization and collision avoidance.
- Conclusion: Edge-FS combines computationally efficient optical flow and stereo processing on a lightweight stereo camera carried by a 40 g pocket drone.The forward-facing camera is designed for environments with complex three-dimensional structure.
- Conclusion: Autonomous experiments in a 4 × 4 meter area showed that velocity estimates and depth measurements supported flight navigation and obstacle avoidance.A simple finite-state-machine controller was used in these experiments.
- Conclusion: The work establishes a basis for stabilization and collision avoidance using a single small stereo-vision system on pocket drones.