PaperlayerView exact revision on arXiv

Source-linked AI summary

Sensor-Layout-Agnostic Navigation via Geometric Observation Canonicalization

Welf Rehberg, Kostas Alexis

arXiv:2610.08306v1cs.RO

TL;DR

Visual navigation policies are often bound to fixed camera layouts, limiting zero-shot deployment across changing sensor payloads. The paper canonicalizes depth feeds into a shared spherical range image and validity mask, then trains with camera randomization. The policy transfers to unseen layouts with up to seven cameras and is validated in physical quadrotor trials.

  • Problem

    Visual navigation policies are commonly tied to fixed camera configurations, so changes to sensor payloads can make them unusable and require retraining.

    Source: §I · p. 1

  • Method

    The policy unprojects arbitrary depth feeds into a shared egocentric spherical range representation and validity mask, and is trained with randomized sensor extrinsics and intrinsics.

    Source: §I · p. 1

  • Results

    The policy generalizes in simulation to unseen layouts with up to seven cameras, performs comparably to layout-specific specialists, and is validated on a quadrotor across sensor configurations and sudden dropouts.

    Source: §I · p. 2

  • Takeaways & Limitations

    The approach supports zero-shot adaptation across heterogeneous depth-camera payloads on the considered platform.

    Source: §VI · p. 7

    The problem considers varying depth-camera setups on a fixed dynamical system.

    Source: §I · p. 1

Abstract

from arXiv · show

Existing visual navigation policies are inherently bound to fixed camera configurations, creating a fundamental barrier to zero-shot deployment across heterogeneous robot sensor layouts. To overcome this limitation, we present an embodiment-informed navigation policy capable of generalizing across diverse depth sensor configurations on a specific aerial platform. Instead of implicitly learning spatial alignments, our approach explicitly unprojects depth measurements from arbitrary depth sensor payloads, varying in sensor count, mounting extrinsics, and intrinsics, into a shared robot-centric frame, stitching them into a unified spherical range image and a binary validity mask. This mask allows the downstream policy to explicitly distinguish covered space from unobserved blind spots. Trained via reinforcement learning with aggressive camera randomization, our policy generalizes zero-shot to unseen layouts featuring up to seven cameras, scaling success rates from 78% to 95% as total spatial sensing coverage increases. Finally, real-world flight trials on a physical quadrotor, conducted in an obstacle-filled corridor and an outdoor forest, validate the policy's zero-shot transfer across camera configurations and its resilience to sudden online sensor dropouts.

I. INTRODUCTION

The paper addresses visual navigation policies’ dependence on fixed camera configurations, which makes sensor-payload changes require costly retraining. It canonicalizes depth observations and trains a generalist policy for zero-shot transfer across layouts, evaluating it in simulation and on a physical quadrotor.

I. INTRODUCTION

  • Existing visual navigation policies are tied to camera configurations and vehicle frames, so even small payload changes can make them unusable and require network retraining.Source: §I · p. 1
  • The method unprojects arbitrary depth feeds into a shared egocentric spherical range representation and binary validity mask, covering variations in sensor count, extrinsics, and intrinsics.The mask distinguishes sensed space from unobserved regions.Source: §I · p. 1
  • Reinforcement-learning training randomizes sensor extrinsics and intrinsics so the policy need not implicitly learn sensor-to-body-coordinate alignments.Source: §I · p. 1
  • The policy generalizes in simulation to unseen layouts with up to seven cameras, with performance comparable to layout-specific specialists.Physical quadrotor trials validate zero-shot transfer across sensor configurations and resilience to sudden online sensor dropouts.Source: §I · p. 2
  • Physical quadrotor trials in cluttered indoor corridors and dense forests demonstrate zero-shot transfer across sensor configurations and resilience to sudden online sensor dropouts.Source: §I · p. 2

II. RELATED WORK

Prior visual generalists typically assume fixed sensor payloads, while the paper targets real-time reactive navigation across heterogeneous depth-camera layouts. The problem is formulated around a family of sensor configurations, with the policy receiving proprioceptive, goal, action-history, and depth-sensor inputs.

II. RELATED WORK

  • Prior visual generalists commonly assume a fixed nominal camera payload, while 1D laser-scan projection restricts navigation to a horizontal plane and multi-view VLA methods target slower planning.The cited VLA approaches also have latency or sensor-layout constraints.Source: §II · p. 2
  • The proposed lightweight generalist targets onboard 3D reactive collision avoidance across arbitrary sensor counts, mounting extrinsics, and intrinsics.It canonicalizes depth feeds into an egocentric spherical range tensor and validity mask.Source: §II · p. 2

III. PROBLEM FORMULATION

  • The problem treats sensor configurations as a system family described by camera extrinsics and intrinsics, with up to ncam arbitrarily placed and oriented depth sensors.Source: §III · p. 2
  • The policy produces body-frame linear reference velocities and yaw rate toward a goal specified by inertial-frame start and goal positions.Source: §III · p. 2
  • At each timestep, inputs include a proprioceptive state estimate, the previous action, and axial depth observations with each sensor’s extrinsics and intrinsics.Source: §III · p. 2

IV. METHOD

The system family varies camera count, mounting extrinsics, and intrinsics on a fixed multirotor. Training assumes nominal dynamics and a centered principal point; real images can be resampled to the training resolution.

A. System Family

  • Camera count is uniformly sampled from 1–4, while camera positions, orientations, and optical parameters are randomized within specified ranges.The resulting fields of view span 50°–131° in azimuth and 26°–142° in elevation.Source: A. System Family · p. 3
  • Camera orientations exclude fully upward- or downward-facing configurations, and the principal point remains fixed at the image center.The aspect ratio, rather than vertical aperture, is randomized to cover narrow and wide fields of view.Source: A. System Family · p. 3
  • Training assumes nominal system dynamics and fixed image resolution, while resampling real images enables deployment with varying camera resolutions.Known spatial offsets are compensated during depth-scan preprocessing.Source: A. System Family · p. 3

1) Canonicalized Spherical Range Image

Depth scans are canonicalized into a robot-centric spherical range image, paired with a mask that distinguishes observed from unobserved directions, and encoded with robot state for the policy.

1) Canonicalized Spherical Range Image

  • Depth pixels are unprojected with camera intrinsics and extrinsics, binned in robot-centric spherical coordinates, and overlapping bins retain the nearest return before inverse-range encoding.When multiple points fall into one bin, the smallest distance is selected; the same rule applies where camera regions overlap.Source: 1) Canonicalized Spherical Range Image · p. 3

2) Validity Mask

  • The binary validity mask marks bins with a projected depth measurement as 1 and unobserved regions as 0.For a static camera configuration, the mask remains constant during evaluation.Source: 2) Validity Mask · p. 3

C. Architecture

  • The network encodes range-plus-mask and robot state, fuses the features, then uses a 64-unit GRU with separate actor and critic heads.The depth encoder has five convolutional layers with circular azimuth padding.Source: C. Architecture · pp. 3–4

D. Training

Training uses rewards for goal progress, collision avoidance, visibility-aligned motion, and action regularization. An auxiliary reconstruction objective encourages the depth encoder to retain information about observed regions.

1) Reward Design

  • The scalar reward combines goal-directed terms with visibility alignment, collision avoidance, and penalties for abrupt or unnecessarily large actions.Goal progress uses two position-based rewards at different spatial scales; collisions that terminate an episode receive a penalty.Source: 1) Reward Design · p. 4
  • The visibility reward scales commanded velocity by the fraction of valid depth bins in an angular window centered on that direction.The collision penalty uses valid measurements and motion toward obstacles to estimate time-to-collision across the range grid.Source: 1) Reward Design · p. 4
  • An auxiliary decoder reconstructs the canonicalized range image and validity mask from depth features, with the validity mask restricting range loss to measured bins.The mask reconstruction uses binary cross-entropy, and reconstruction losses contribute alongside PPO loss to network optimization.Source: 1) Reward Design · pp. 4–5

2) Environment Setup

Training uses a randomized obstacle-filled room and a performance-based curriculum, alongside varied camera and robot initial conditions.

2) Environment Setup

  • The training room contains randomly positioned and oriented obstacles spanning small thin objects to large flat panels.Obstacle density increases when the policy reliably reaches the goal at the current difficulty level.Source: 2) Environment Setup · p. 5
  • Each reset samples a random number of depth cameras and randomizes the robot's initial state and goal position.Source: 2) Environment Setup · p. 5
  • Stereo-depth noise and randomized low-level controller parameters are used during training to support sim-to-real transfer.Source: 2) Environment Setup · p. 5

V. EVALUATION

The evaluation combines simulation trials and real-world experiments, with simulation performance measured over resampled system configurations and obstacle layouts.

V. EVALUATION

  • Evaluation policies are tested for 1,000 episodes with system configuration and obstacle layout resampled at every reset.Simulation training used 1.9 × 10^6 environment steps, and evaluation maintains the curriculum at maximum difficulty.Source: §V · p. 5
  • At maximum curriculum difficulty, simulated environments contain 37 randomly placed obstacles.Source: §V · p. 5
  • Crash rate and success rate are reported; success requires reaching within 1 m of the goal, while collision counts as a crash.Episodes meeting neither condition are timed out.Source: §V · p. 5

A. Ablation Studies

The ablations test the binary validity mask and auxiliary masked reconstruction loss, which the authors report make complementary contributions to policy performance.

A. Ablation Studies

  • Both the validity mask and auxiliary masked reconstruction loss contribute to reducing crash rate and maximizing success rate across the training population.Source: A. Ablation Studies · p. 5
  • One baseline removes both the mask from policy input and the masked reconstruction loss; another retains the mask input but disables reconstruction loss.In both baselines, the mask remains in the reward calculation.Source: A. Ablation Studies · p. 5
  • Adding the mask and reconstruction loss increases sample efficiency and training stability in the ablated-method curriculum curves.Source: A. Ablation Studies · p. 5

B. Simulation Results

Simulation tests evaluate performance across camera counts, specialist comparisons, and a dense two-camera design space. The policy generalizes to unseen camera counts and approaches specialist performance, while camera mounting angles affect outcomes.

B. Simulation Results

1) Performance Evaluation

  • As camera count rises, CR decreases and SR increases before plateauing; the authors hypothesize this tracks combined field-of-view coverage.The 5-, 6-, and 7-camera layouts were unseen during training; additional sensors eventually provide no novel or actionable information.Source: 1) Performance Evaluation · p. 5

2) Comparison to Specialist Policies

  • The generalist incurs only a 2% increase in CR and a 2% decrease in SR versus specialists trained on fixed camera configurations.The comparison averages results across eight configurations spanning camera counts of one through four.Source: 2) Comparison to Specialist Policies · p. 6, Table IV · p. 6
  • Against the specialized single-camera architecture, the generalist performs on par across six configurations; its comparison with is approximate and reports a 6% lower CR for.The comparison uses results from a different implementation and environment; the authors suggest its explicit FOV constraints may explain the lower crash rate.Source: 2) Comparison to Specialist Policies · p. 6, Table V · p. 6

3) Design Space Evaluation

  • Across a dense sweep of two-camera layouts, performance worsens as pitch departs from the body’s x−y plane and at extreme relative yaw angles.The authors attribute the pitch effect to underactuated dynamics and the yaw effects to redundant FOV overlap or reduced forward-flight coverage.Source: 3) Design Space Evaluation · p. 6, Figure 7 · p. 6

4) Sensitivity Analysis

The sensitivity analysis tests how embodiment-parameter mismatches affect collision rate and success rate. Performance remains stable within parameter-specific disturbance bounds, with tighter tolerance for camera orientation than for intrinsic parameters or position.

4) Sensitivity Analysis

  • Fig. 8 plots CR and SR as embodiment parameters used for canonicalization are perturbed by increasing amounts.Perturbation magnitudes are percentages of each parameter’s randomization range, with disturbance signs sampled equally.Source: 4) Sensitivity Analysis · p. 6, Figure 8 · p. 6
  • Performance does not drop for mismatches up to 10% of the randomization range for intrinsic parameters or camera position.The tested range is relative to the parameter randomization range, not an absolute calibration error.Source: 4) Sensitivity Analysis · p. 6
  • Camera-orientation tolerance is lower: yaw mismatches remain robust up to 2.5% and roll and pitch mismatches up to 7.5% of their respective randomization ranges.The authors cite these results as support for suitability in sim-to-real transfer.Source: 4) Sensitivity Analysis · p. 6

C. Real-World Experiments

Physical quadrotor trials test the policy across sensor configurations in a cluttered hallway and dense forest. The trials show obstacle avoidance, adaptation to sensor changes, and navigation through complex natural geometry without fine-tuning.

C. Real-World Experiments

  • The quadrotor runs the policy onboard at 10 Hz and is tested with three, two, or one active D455 camera in a cluttered corridor and dense forest.The platform uses onboard radar-inertial odometry and PX4 velocity tracking.Source: C. Real-World Experiments · p. 7

1) Cluttered Hallway

  • In the hallway, the policy avoids obstacles and, when its only active camera faces away from the goal, yaws in place before moving forward.For a second dropout scenario, it halts forward motion, turns about 180° toward a newly activated opposite-facing camera, then resumes navigation.Source: 1) Cluttered Hallway · p. 7
  • Replacing a D455 with a narrower-FOV pmd flexx2 ToF sensor supports successful navigation without retraining.The setup combines the pmd flexx2 with a D455 and tests adaptation to different sensor models, resolutions, and fields of view.Source: 1) Cluttered Hallway · p. 7

2) Forest

  • In the forest, the policy navigates thin trunks, branches, and irregular foliage without fine-tuning, using multiple sensor streams for smooth 3D avoidance.Vertical velocity is unconstrained in this environment.Source: 2) Forest · p. 7

VI. CONCLUSION AND FUTURE WORK

  • The conclusion reports zero-shot simulation generalization to payloads with up to seven cameras and real-world resilience to complex geometries and camera dropouts.Joint transfer across sensor layouts and physical robot platforms is left to future work.Source: §VI · p. 7
Loading 2610.08306v1…