PaperlayerView exact revision on arXiv

Source-linked AI summary

TinyCVIO: A Constellation-Aided Visual-Inertial Odometry System for Nanodrones

Derin Ozturk, Kaan Akan, Irwin Wang, Christopher Batten

arXiv:2609.30358v1cs.RO

TL;DR

Nanodrones need accurate real-time state estimation despite tight sensing and compute budgets. TinyCVIO combines LED constellations with visual-inertial estimation and streaming computation; on identical recorded inputs, its rigid-board model reduces mean ATE by 27% relative to planar points.

Abstract

from arXiv · show

Nanodrones require accurate, real-time state estimation under severe sensing and computational constraints. We present TinyCVIO, a visual-inertial odometry system that co-designs miniature sensing, visual processing, and estimation for a commodity dual-core microcontroller with 520 kB SRAM. Lightweight LED constellations provide known geometry without surveyed positions or yaw angles, assuming placement on a common level plane. A streaming visual frontend tracks LED observations from a millimeter-scale camera at 29.2 FPS, while a rigid-board measurement model retains inter-LED constraints and streaming QR bounds estimation workspace for a fixed filter-state size. Across 19 hand-held hardware-in-the-loop datasets, the rigid-board model reduces mean absolute trajectory error by 27% relative to planar points. The complete system runs onboard a Crazyflie across nine flights at three speeds, achieving 3.5-3.7 cm mean absolute trajectory error and 0.50-0.60% relative pose error over 10 m segments, with mean estimate latency of 15.7-16.3 ms.

I. INTRODUCTION

TinyCVIO addresses nanodrone position drift while keeping onboard resource demands, system complexity, and deployment effort low. It combines LED constellations with visual-inertial estimation on a commodity microcontroller.

A. Localization and Onboard Sensing

  • Minimal optical-flow and inertial sensing supports hover but lacks an absolute horizontal position reference, while infrastructure-based localization requires equipment, installation, and spatial calibration.Richer onboard approaches use larger compute budgets, companion processors, custom silicon, or ground-station processing.Source: §I · p. 1, A. Localization and Onboard Sensing · p. 2

I. INTRODUCTION

  • The system jointly designs miniature sensing, visual processing, and estimation around LED constellations and a commodity RP2354 dual-core microcontroller.The contributions include a geometry-aware rigid-board MSCKF model, streaming estimation, and evaluation through simulation, hardware-in-the-loop experiments, replays, and flight.Source: §I · p. 1
  • LED constellations provide known geometric relationships without surveying board positions or yaw angles, assuming placement on a common level plane.The geometry supplies visually distinctive landmarks and constraints for estimation.Source: §I · p. 1

B. Structured Landmarks and Pose Estimation

  • TinyCVIO fuses visual and inertial measurements over time, unlike prior structured-landmark pose-only approaches.The cited prior MCU system used planar LED landmarks with known positions and orientations; TinyCVIO instead exploits known within-constellation geometry and eliminates shared board pose through nullspace projection.Source: B. Structured Landmarks and Pose… · p. 2

C. Embedded Visual-Inertial Odometry

TinyCVIO pairs miniature sensing with a commodity dual-core MCU for onboard nanodrone VIO. Its hardware, LED landmarks, acquisition pipeline, and concurrent execution are designed for constrained deployment.

C. Embedded Visual-Inertial Odometry

  • Prior embedded VIO systems establish MCU or GAP9 feasibility, but the cited evaluations do not report a complete pipeline operating onboard a nanodrone in flight.The described prior systems include Cortex-M7 execution with 3 MB RAM and recent GAP9 implementations.Source: C. Embedded Visual-Inertial Odometry · p. 2

III. SYSTEM DESIGN

  • A custom nanodrone PCB combines a monocular camera and IMU with dual-core processing, assigning visual frontend and visual-inertial backend work to separate cores.The RP2354 runs at 300 MHz with 520 kB SRAM; the camera operates at 29.2 FPS and the IMU at 400 Hz.Source: §III · p. 2, A. Hardware and Constellations · p. 2

A. Hardware and Constellations

  • Each battery-powered constellation uses four LEDs on a 40 mm square and a fifth disambiguator, with known geometry on a common level plane but unsurveyed positions and yaw angles.The fifth LED breaks rotational symmetry.Source: A. Hardware and Constellations · p. 2

B. Streaming Sensor Acquisition

  • Programmable I/O extracts 8-bit pixels from 12-bit camera packets and generates threshold events, while a statically reserved half-image buffer supports HITL streaming but is unnecessary for onboard estimation.PIO framing, extraction, and thresholding occupy all 32 instruction slots in one block.Source: B. Streaming Sensor Acquisition · p. 2

C. Concurrent Execution

  • Core 0 handles acquisition and visual tracking while Core 1 propagates inertial state and performs updates; static allocation and a fixed SRAM arena reserve backend matrix capacity.Propagation is spread across the inter-frame interval, and the estimator consumes the latest visual frame without blocking the frontend.Source: C. Concurrent Execution · pp. 2–3, Figure 2 · p. 2

IV. VISUAL FRONTEND

The visual frontend turns threshold events into labeled LED observations, maintaining constellation correspondences as viewing distance changes and LED images fragment or merge.

IV. VISUAL FRONTEND

  • Known constellation geometry and temporal tracking maintain LED correspondences despite changing apparent board size, fragmentation, and merged LED components.The pipeline outputs image coordinates and board and LED identifiers.Source: §IV · p. 3, Figure 3 · p. 3

A. Event Clustering

  • A single raster-order pass forms candidate constellation groups and connected components, using adaptive grouping radii and a two-row scanline buffer to compute component statistics without retaining a full image.Groups are finalized once readout advances beyond their spatial extent.Source: A. Event Clustering · p. 3

B. Association and Identification

  • New constellations are identified by fitting candidate LED assignments to known board geometry, while second moments can split a component containing two merged LEDs.Up to 30 assignments are ranked by a closed-form similarity fit; IPPE selects the front-facing pose and lowest five-point reprojection-error assignment.Source: B. Association and Identification · p. 3

C. Tracking and Label Continuity

Tracking preserves LED labels across frames using fitted image transforms and geometry checks. The backend uses an MSCKF-style state and streaming QR strategy to bound estimation workspace.

C. Tracking and Label Continuity

  • A fitted 2D similarity transform propagates LED correspondences across frames, including merged-pair positions, with nearest-neighbor matching and re-identification as fallbacks.A continuity check rejects configurations whose square orientation changes beyond 30° or whose chirality reverses.Source: C. Tracking and Label Continuity · p. 3

V. VISUAL-INERTIAL BACKEND

  • The backend follows an OpenVINS-style separation of measurement modeling, propagation, nuisance elimination, compression, and filter update within the MSCKF framework.IMU and camera-pose clone states are retained, but LED positions and board poses are not; optional time-offset calibration adds a scalar state.Source: §V · p. 3, A. State and MSCKF Overview · pp. 3–4

A. State and MSCKF Overview

  • Streaming QR folds projected track constraints into a running factor without assembling the full measurement stack, discarding zero-Jacobian residuals when rows exceed the factor's column dimension.Storage is bounded by an O(n^2 state) running factor plus one track's incoming rows, independently of track count.Source: A. State and MSCKF Overview · p. 4
  • A very small per-update track cap reduces streaming's memory benefit across tracks, although streaming board-pose elimination still bounds the cross-frame accumulator.The optional cap prioritizes the longest tracks.Source: A. State and MSCKF Overview · p. 4

B. Rigid-Board Measurement Co-Design

The rigid-board model uses known LED geometry and shared planar placement to preserve constellation structure in visual-inertial measurements. Each board is represented by a constrained pose, while observations are linearized with respect to that pose and the camera clone pose.

B. Rigid-Board Measurement Co-Design

  • Known LED positions in a board-fixed frame share a board pose, exploiting inter-LED geometry under common known-plane placement.The model also applies to other rigid fiducials with known point geometry placed on a common known plane.Source: B. Rigid-Board Measurement Co-Design · p. 4
  • Known level-plane placement restricts the board-to-global transformation to two translations and one yaw angle.The translation has zero vertical component, and the board-pose nuisance parameter is β = (tx, ty, θ).Source: B. Rigid-Board Measurement Co-Design · p. 4
  • Each LED observation at a camera clone is modeled with Jacobians for the shared board pose and the clone pose.The clone pose is stored in the state, and Hβ is obtained by differentiating the board-geometry relation.Source: B. Rigid-Board Measurement Co-Design · p. 4

C. Board-Pose Recovery and Nullspace Elimination

The method recovers a shared board pose from multiple clone frames, then eliminates that nuisance pose with streaming QR while retaining board-induced state constraints. Local compression bounds the rows entering the shared fold for boards with at least five LEDs.

C. Board-Pose Recovery and Nullspace Elimination

  • IPPE recovers a board-to-camera pose per clone frame, and the lowest-reprojection-error frame seeds a joint Gauss-Newton refinement over (tx, ty, θ).Optional Cauchy IRLS weighting and χ2 gating suppress outliers; robust weights whiten residual and Jacobian rows before nullspace projection.Source: C. Board-Pose Recovery and Nullspace… · p. 4
  • Two-level streaming QR eliminates the shared board pose while preserving inter-LED constraints in the resulting state constraints.Local compression assumes fixed camera intrinsics and extrinsics, with no additional calibration columns.Source: C. Board-Pose Recovery and Nullspace… · p. 4
  • Per-frame Givens rotations triangularize the board-pose Jacobian and compress board-free rows over the clone’s six pose columns.The top three board-coupled rows are retained for cross-frame accumulation, while middle board-free rows enter the shared fold.Source: C. Board-Pose Recovery and Nullspace… · p. 4
  • A three-row accumulator combines board-coupled rows across frames, and discarding it after the final frame eliminates the board nuisance pose.The bottom three board-free rows enter the shared fold for accepted tracks at each accumulation step.Source: C. Board-Pose Recovery and Nullspace… · pp. 4–5
  • For Nℓ≥5, local compression limits rows passed to the shared fold to 9F −3, independent of board size, while measurement evaluation still scales with LED count.The cross-frame elimination maintains a three-row accumulator.Source: C. Board-Pose Recovery and Nullspace… · p. 5

D. Sensor Compensation and Initialization

The system compensates for rolling-shutter timing without adding state variables and provides several initialization options. Its measurement-model comparison contrasts the rigid-board model with independent-point models using the same LED observations and downstream update.

D. Sensor Compensation and Initialization

  • Each board observation receives an effective capture time from its mean readout row, and the clone pose is extrapolated using frozen motion estimates without augmenting the state.This compensates for the NanEyeC rolling shutter, whose image rows are captured at different times.Source: D. Sensor Compensation and… · p. 5
  • On-drone pad initialization recovers initial pose from LED observations, gravity, and known camera height, while a static IMU window supplies biases.The 3D-printed pad defines the navigation frame; stationary updates maintain the estimate before takeoff.Source: D. Sensor Compensation and… · p. 5

E. Measurement-Model Comparison

  • The comparison uses the same LED observations and streaming update for a rigid-board model, a 3D independent-point MSCKF model, and a planar-point model estimating only horizontal coordinates.Neither point model retains known inter-LED geometry; planar-point recovery uses non-grazing ray–plane intersections rather than translational parallax.Source: E. Measurement-Model Comparison · p. 5
  • Table I compares track counts, nuisance degrees of freedom per track, and rows supplied to the shared fold for processing one board’s LEDs over F frames.The row counts assume full-rank nuisance Jacobians and do not represent independent observable state directions.Source: E. Measurement-Model Comparison · p. 5, Table I · p. 5

VI. EXPERIMENTAL METHODOLOGY

The evaluation combines controlled simulation, hand-held hardware-in-the-loop captures, and onboard experiments. Simulation varies trajectories and modeling conditions, while the shared testbed uses calibrated sensing and Vicon timing and motion capture.

VI. EXPERIMENTAL METHODOLOGY

  • The evaluation combines controlled simulation, hand-held hardware-in-the-loop captures, and on-drone experiments.These settings are used to evaluate the proposed system.Source: §VI · p. 5

A. Simulation Protocol

  • Seven Webots figure-eight trajectories span 0.35–0.99 m s−1 mean speed, up to 19° tilt, and 0.7–1.2 m altitude, with synthetic camera and IMU measurements at 30 Hz and 400 Hz.All visible boards are reported with correct labels to isolate backend performance from detection and association failures; the nominal board layout and LED pattern are fixed.Source: A. Simulation Protocol · p. 5
  • Simulation reports median 4-DOF-aligned ATE across seven equally represented trajectories, retaining runs above 50 cm ATE and also reporting their divergence rate.Each tested parameter value uses 50 noise and perturbation seeds per trajectory; sweeps include detection noise, placement and ground-plane errors, camera errors, clone-window length, gating, and measurement budget.Source: A. Simulation Protocol · p. 5

B. Hardware and Shared Testbed

  • The hand-held rig combines a Vicon-marked carrier with a Raspberry Pi Pico 2, NanEyeC camera, and ICM-45686 IMU; its sensing and compute architecture matches the drone platform, using an RP2350 with external flash instead of an RP2354.Experiments use Vicon and constellation boards on flat surfaces in 2×2 to 4×4 grids at approximately 0.61 m spacing; camera frame-sync and Vicon timing are aligned below one millisecond.Source: B. Hardware and Shared Testbed · pp. 5–6
  • Camera intrinsics, camera–IMU extrinsics, and the Vicon-to-sensor transform are calibrated using AprilGrid, Kalibr, and hand–eye calibration, respectively.Source: B. Hardware and Shared Testbed · p. 6

C. HITL Evaluation

The evaluation measures frontend accuracy and detection behavior, then describes the live and onboard evaluation protocols. In frontend tests, localization is accurate overall, while detection and label continuity remain important constraints.

C. HITL Evaluation

  • Live HITL runs execute the frontend and estimator on the microcontroller; paired host replays use identical recorded observations, initialization, calibration, and LED budgets to compare models.Estimates are scored against Vicon after SE(3) alignment using position and orientation RMSE and translational RPE over 10 m segments; detection is assessed against reference LED projections.Source: C. HITL Evaluation · p. 6

D. On-Drone Evaluation

  • The end-to-end evaluation includes onboard Crazyflie flights with TinyCVIO estimates sent over I2C and scored during the airborne window against Vicon.The deck publishes poses at 50 Hz; flight scoring uses a fitted mounting rotation for orientation error and a fitted temporal offset per flight.Source: D. On-Drone Evaluation · p. 6, §VII · p. 6

A. Frontend Evaluation

  • 0.6 px overall LED localization RMS error varies little across tested distances and angular velocities, but increases near the image border.Figure 4 bins frontend performance by angular velocity, board distance, and image radius; its rows show localization error, mislabeled detections, detected and backend-accepted boards, and board counts.Source: A. Frontend Evaluation · p. 6, Figure 4 · p. 6
  • The frontend detects 82% of in-view boards within 1.4 m and 140 px of image center; missed boards can remain bright as LEDs merge or become difficult to distinguish.Among missed boards, 91% remain bright, so brightness alone does not ensure detection.Source: A. Frontend Evaluation · p. 6
  • The label-continuity guard cuts mislabeled detections from 22.8% to 4.9% and raises the share entering accepted backend updates from 43% to 52%.Errors typically begin as mid-track label flips after merged-LED observations; suppressing these flips preserves more observations for accepted updates.Source: A. Frontend Evaluation · p. 6

B. Backend Evaluation

Backend evaluation compares geometric constraints, model assumptions, robustness settings, and measurement budgets. The rigid-board model has the lowest median ATE across tested detection-noise levels, with trade-offs from calibration error and workspace limits.

Measurement model

  • Rigid-board measurements achieve the lowest median ATE across tested detection-noise levels, with an eightfold advantage over free points at 0.25 px noise and a threefold advantage at 2 px.As detection noise decreases, rigid-board accuracy continues improving while free-point accuracy reaches a floor; increasing the clone window from six to sixteen frames reduces ATE by 35% for rigid boards, versus 25% for planar points and 17% for free points.Source: B. Backend Evaluation · p. 7

Model assumptions

  • A 2 mm LED placement error raises rigid-board ATE by 66%, while changing either point model’s error by less than 3%.Ground-plane height variation affects both plane-constrained models more than free points; 1.5% distortion error raises ATE by 17–27% across models without changing their ranking.Source: B. Backend Evaluation · p. 7

Gate and weighting

  • With uniform weighting, a χ2 gate multiplier of 0.5 makes the rigid-board model diverge in 82% of runs; with Cauchy weighting, it has no observed divergence for tested multipliers from 0.5 to 20.Source: B. Backend Evaluation · p. 7

Measurement budget

  • Limiting updates to one board reduces fold workspace to one-quarter of the uncapped requirement while increasing ATE by approximately 10%.Across tested detection-noise levels, its ATE stays 9–11% above the uncapped result; at matching noise it remains more accurate than either independent-point model at any tested budget.Source: B. Backend Evaluation · p. 7

C. HITL Evaluation

In paired HITL replays, the rigid-board model improves trajectory accuracy over planar points; repeated onboard flights show consistent position accuracy and stable estimate latency. TinyCVIO combines known constellation geometry with streaming estimation to support onboard nanodrone state estimation.

C. HITL Evaluation

  • 27% lower mean ATE with the rigid-board model than planar points on identical inputs, with lower ATE on 18 of 19 captures and lower mean orientation and relative translation errors.Table III reports mean errors over 19 paired capture replays; comparisons use identical recorded frontend observations and the same LED budget.Source: C. HITL Evaluation · p. 7, Table III · p. 7

D. On-Drone Demonstration

  • Fig. 6 shows estimated and Vicon trajectories for one fastest-group flight after SE(3) alignment.Source: Figure 6 · p. 8, D. On-Drone Demonstration · p. 8
  • Across three figure-eight flights at mean ground-truth speeds of 0.34, 0.41, and 0.50 m s−1, position accuracy remained consistent, with greater variability in the fastest group, while estimate latency stayed stable for real-time onboard estimation.Source: D. On-Drone Demonstration · pp. 7–8

VIII. CONCLUSION

  • TinyCVIO combines known constellation geometry with streaming QR to improve estimation accuracy while bounding workspace for a fixed state size.The conclusion reports simulation, paired replays, live hardware experiments, and repeated Crazyflie flights as demonstrations of the design.Source: §VIII · p. 8
Loading 2609.30358v1…