PaperlayerView exact revision on arXiv

Source-linked AI summary

Object- and Task-Aware Active Perception for Safe Navigation of Mobile Manipulators in Semi-Static Environments

Jingxing Qian, Xintong Du, Jun Yang, Siqi Zhou, Steven L. Waslander, Angela P. Schoellig

arXiv:2610.29964v1cs.RO

TL;DR

Clutter and semi-static changes can make map knowledge outdated in regions important to safe navigation. The paper uses object- and task-aware active perception, coordinated with navigation-first control, and reports improved map consistency, lower task-dependent map entropy, and lower navigation risk than passive or non-task-aware baselines.

  • Problem

    In semi-static environments, outdated knowledge in path-relevant and safety-critical regions can increase collision risk, while passive mapping may leave occluded areas unobserved.

    Source: §I · pp. 1–2

  • Method

    A probabilistic object- and task-aware NBV objective prioritizes uncertain or potentially changing regions along the planned path, with HTMPC assigning navigation higher priority than camera tasks.

    Source: §I · p. 3

  • Results

    Real-world experiments report reduced task-relevant map uncertainty, improved map consistency, and lower navigation risk than passive or non-task-aware baselines.

    Source: §VIII · p. 17

  • Takeaways & Limitations

    The results support prioritizing uncertain, potentially changing regions according to their relevance to safe progress along the planned path.

    Source: §VIII · p. 17

Abstract

from arXiv · show

Mobile manipulators must navigate safely in cluttered, partially unknown environments across logistics, manufacturing, and service applications. Semi-static changes between robot visits can lead to outdated scene knowledge and increased collision risk. We propose an active perception framework that enables a high degree-of-freedom (DoF) mobile manipulator to inspect uncertain or safety-critical regions with its hand camera during navigation. A probabilistic, object- and task-aware next-best-view (NBV) formulation prioritizes observations near the desired path and around uncertain or potentially changing objects. The perception objective is integrated into hierarchical-task model predictive control (HTMPC), exploiting kinematic redundancy for viewpoint adjustment while preserving the higher priority of base-trajectory tracking. The system unifies navigation, map maintenance, and active perception in a closed loop. Real-world tests show improved map consistency, reduced task-dependent map entropy, and lower navigation risk than passive or non-task-aware baselines. Additional simulation and real-world sequences demonstrate responses to controlled semi-static scene changes.

I. INTRODUCTION

The introduction frames safe navigation in semi-static, cluttered environments as a map-maintenance problem: passive sensing can miss occluded changes, especially in regions that matter to the planned path. It proposes object- and task-aware active perception integrated with navigation-first control.

I. INTRODUCTION

  • Clutter and between-visit scene changes can leave occluded regions outdated, delaying obstacle detection and increasing collision risk or reducing navigation efficiency.The introduction highlights path-adjacent safety-critical areas, narrow passages, and blind corners as particularly consequential regions.Source: §I · pp. 1–2

C. Active Perception in Robotics

  • Prior NBV approaches often target short-horizon static scenes or geometric completeness, rather than jointly addressing long-term scene changes, object-level consistency, and current navigation-task relevance.The proposed strategy directs attention to uncertainty that may affect path clearance and the next maneuver.Source: §I · p. 2

I. INTRODUCTION

  • The proposed framework uses a probabilistic NBV objective to prioritize uncertain or potentially changing regions along the planned path, while HTMPC preserves navigation priority during camera repositioning.The robot uses manipulator redundancy to inspect uncertain, occluded, and safety-critical regions with its hand camera.Source: §I · p. 3
  • In tested static scenes, the system reduces task-dependent map entropy, improves map consistency, and lowers navigation risk versus passive or non-task-aware baselines; additional sequences show closed-loop responses to scene changes.The framework combines path-conditioned map uncertainty with hierarchical control that gives navigation priority over camera tasks.Source: §I · p. 3

A. Semantic and Object-Aware Mapping

  • Object-aware mapping adds semantic and object-level reasoning to geometric maps, but many systems prioritize geometric accuracy under a static-world assumption and can accumulate inconsistencies over long deployments.Such inconsistencies can degrade localization and increase safety risks.Source: A. Semantic and Object-Aware Mapping · p. 3

B. Change Detection in Semi-Static Environments

  • Semi-static change-detection methods can improve long-term map consistency, but are typically evaluated with passive observations, leaving occluded or task-critical changes potentially undetected until failure.The introduction identifies online viewpoint selection and control coupled to probabilistic object-level change estimates as an open question.Source: B. Change Detection in Semi-Static… · p. 3

D. Perception-based Mobile Manipulation

The paper treats active perception as an explicit, lower-priority task in hierarchical control, coordinating sensor placement with navigation. Its semi-static map represents object-level geometric change and consistency probabilistically, updating those beliefs with geometric and semantic evidence.

D. Perception-based Mobile Manipulation

  • HTMPC makes active perception an explicit task below navigation, using kinematic redundancy to optimize sensor placement while resolving conflicts between perception and control.The resulting loop plans motion from the latest perceived map while also optimizing sensor placement.Source: D. Perception-based Mobile Manipulation · p. 4

A. Object-Aware Semi-Static Mapping

  • Each mapped object has a geometric-change variable and a consistency belief, allowing the map to reason about object validity across repeated visits and partial observations.The consistency belief is parameterized with a Beta distribution, while geometric change uses a Gaussian distribution.Source: A. Object-Aware Semi-Static Mapping · p. 4
  • Object-observation updates combine geometric consistency evidence with a semantic prior about whether the object is likely static or changing.The geometric likelihood models consistent observations as inliers and changes as outliers; the Bayesian posterior combines new evidence with the previous state estimate.Source: A. Object-Aware Semi-Static Mapping · p. 4
  • Matched observations that agree with the object model typically raise estimated consistency and reduce uncertainty, while inconsistent or missing observations penalize consistency and increase uncertainty.When expected consistency falls below a threshold, the object is treated as changed and reconstructed in the maintained map.Source: A. Object-Aware Semi-Static Mapping · pp. 4–5

B. Fisher Information and Differential Entropy for NBV

The section frames next-best-view planning as reducing task-relevant uncertainty, then uses hierarchical-task model predictive control to realize camera viewpoints without sacrificing higher-priority navigation.

B. Fisher Information and Differential Entropy for NBV

  • The Fisher information matrix measures local observation sensitivity to a latent quantity, while differential entropy measures remaining uncertainty under a local Gaussian approximation.Under that approximation, estimate covariance is approximated by the inverse Fisher information matrix.Source: B. Fisher Information and Differential… · p. 5
  • Approximate expected information gain ranks viewpoints by predicted reduction in task-dependent map entropy, averaging over weighted grid cells, rollout times, and two representative observation outcomes.Realizing a selected view also requires coordinating camera and base motion.Source: B. Fisher Information and Differential… · p. 5

C. Hierarchical-Task Model Predictive Control

  • HTMPC orders task costs lexicographically over a finite prediction horizon, subject to robot dynamics and state- and input-dependent constraints such as safety constraints.The optimization is solved in receding-horizon fashion.Source: C. Hierarchical-Task Model Predictive… · p. 5
  • Unlike a single weighted objective, sequential task-cost constraints prevent lower-priority optimizations from increasing higher-priority costs, preserving base tracking while residual redundancy serves camera motion.Inspecting a blind corner can require hand-camera motion around an obstacle while the base follows its path.Source: C. Hierarchical-Task Model Predictive… · p. 5

IV. SYSTEM DESCRIPTION

The system closes the loop among robot-state estimation, object-aware map maintenance, path planning, next-best-view selection, and control. A base camera and hand camera support navigation and viewpoint adjustment on a mobile manipulator.

A. System Overview

  • The five-module pipeline synchronizes measurements, updates the map, plans a base path, scores hand-camera viewpoints by predicted task-dependent map-entropy reduction, and closes the loop with new observations.The control module tracks the base path while realizing the selected hand-camera objective.Source: A. System Overview · p. 6
  • The platform combines a UR10 arm with a Ridgeback omnidirectional base and synchronized RGB-D cameras mounted on the base and hand.An onboard laptop runs perception, planning, and control algorithms.Source: Figure 4 · p. 6

B. Robot State and State Estimation

  • Base pose and velocity are estimated using modified ORB-SLAM3 with base-camera data and wheel odometry, while forward kinematics estimates hand-camera pose from arm joint states.Wheel odometry supports feature-sparse regions and compensates for RGB-D frame-rate and latency limits.Source: B. Robot State and State Estimation · p. 6

C. Object-Aware Perception and Map Maintenance

  • RGB-D observations are segmented into object point clouds and associated with a mapped object library; Bayesian updates maintain object states and uncertainty for downstream viewpoint planning.Objects below a consistency threshold are treated as changed and reconstructed, while stale objects gain synthetic outlier evidence and become more uncertain.Source: C. Object-Aware Perception and Map… · pp. 6–7

D. Object- and Task-Aware Next-Best-View Planning

The NBV module scores hand-camera viewpoints by expected information gain over map regions weighted for uncertainty and relevance to the planned path. HTMPC then realizes selected views while giving base-path tracking higher priority.

D. Object- and Task-Aware Next-Best-View Planning

  • The NBV module ray traces cells visible from sampled poses and scores each pose by approximate expected information gain, weighted by predicted base motion along the path.The maintained map, robot state, and desired base path are inputs; the selected pose becomes the hand-camera reference.Source: D. Object- and Task-Aware… · p. 7

E. Whole-Body Control

  • HTMPC gives base-path tracking higher priority than camera motion, using remaining kinematic redundancy for the selected view while enforcing map-derived collision constraints.Both base and camera references are updated as the map changes.Source: E. Whole-Body Control · p. 7

V. NEXT-BEST-VIEW FORMULATION

  • Task-dependent map entropy is the prior for approximate expected-information-gain scoring, combining map uncertainty with path-conditioned relevance.The formulation constructs map uncertainty and a desired-path prior, then predicts map updates for candidate views.Source: §V · p. 7

A. Map and Uncertainty Construction

  • Object-level consistency links visible changes, such as a shifted cabinet edge, to uncertainty in the object's occluded associated region, prompting inspection of the rest of the object.Cells near mapped objects inherit object-consistency variance; stale objects accumulate synthetic outlier measurements that lower consistency without immediate removal.Source: A. Map and Uncertainty Construction · p. 7
  • The merged 3D map supports whole-body collision avoidance, while a 2D base-height slice supplies path-planning distances and uncertainty for NBV selection.Object-associated cells use consistency variance; other cells use occupancy uncertainty, with an effective EDF-uncertainty model based on these variances as monotone surrogates.Source: A. Map and Uncertainty Construction · pp. 7–8

B. Desired Path and Robot Position Prior

The desired path supplies spatial task-relevance weights for viewpoint selection, but the robot may deviate from it during execution. The formulation models these deviations as a stochastic position rollout.

B. Desired Path and Robot Position Prior

  • The robot position is modeled as a stochastic rollout around the desired path to capture tracking error, collision avoidance, and replanning effects.This rollout weights spatial task relevance for NBV; HTMPC uses a separate, shorter prediction horizon for control.Source: B. Desired Path and Robot Position Prior · p. 8
  • The rollout starts from nominal velocity derived from the global planner's desired path and models executed velocity as deviating from that nominal motion.The resulting base-position rollout is used to construct the position prior.Source: B. Desired Path and Robot Position Prior · p. 8
  • A Gaussian position model centered on the desired path is discretized over the grid and normalized, with a uniform prior over rollout steps.The same grid is used for every candidate view.Source: B. Desired Path and Robot Position Prior · p. 8

C. Task-Dependent Map Entropy

Task-dependent map entropy measures uncertainty where the robot is likely to travel, using the maintained map's distance field and uncertainty. Candidate views are scored by their approximate expected information gain against this prior.

C. Task-Dependent Map Entropy

  • The observation model treats the unknown as a potential robot base position and the observation as the distance-field value, with local Gaussian noise and a finite-difference map gradient.The field's Fisher information is rank one along the local gradient direction; zero-gradient cells are omitted.Source: C. Task-Dependent Map Entropy · p. 9
  • Map entropy increases with larger cell uncertainty and smaller distance-field gradient, reflecting weaker information from a distance observation where the field varies little.Free-space gradients are clipped from above to avoid overconfidence and encourage re-observation for newly added obstacles.Source: C. Task-Dependent Map Entropy · p. 9
  • Task-dependent map entropy is the expectation of cell entropy over path-conditioned grid-cell and rollout-time distributions, and serves as the prior for approximate expected-information-gain scoring.The accumulated hand-camera view history conditions the maintained map belief.Source: C. Task-Dependent Map Entropy · pp. 8–9

D. Candidate Views and Visible-Cell Set

The planner samples candidate hand-camera poses and ray traces their visible cells. For each candidate, it predicts uncertainty reduction using representative synthetic observations and a hypothesized geometric update within the view.

D. Candidate Views and Visible-Cell Set

  • Each candidate viewpoint is a sampled hand-camera target pose, and camera orientation is enforced more strongly than position to permit local adjustment near obstacles.The implementation is path-guided.Source: D. Candidate Views and Visible-Cell Set · p. 9
  • Ray tracing on the maintained map identifies the cells expected to be visible from each candidate, and only those cells receive predicted map updates.The prior entropy is defined over the full grid, but the candidate update is limited to its visible-cell set.Source: D. Candidate Views and Visible-Cell Set · p. 9

E. Predicted Map Update Under a Candidate View

  • For visible object cells, the planner synthesizes inlier and outlier measurements and retains the posterior consistency variance with the smaller value as an optimistic uncertainty-reduction estimate.All visible cells associated with an object inherit that predicted variance.Source: E. Predicted Map Update Under a… · p. 9
  • Within a candidate's field of view, a thin layer of free cells at its frontier is hypothesized occupied to recompute the distance field and gradient; outside it, the current field is retained.Visible non-object cells receive synthetic occupied/free measurements and retain the smaller predicted occupancy variance.Source: E. Predicted Map Update Under a… · pp. 9–10

F. Expected Information Gain and NBV Selection

The approximate expected information gain (EIG) scores each candidate viewpoint by its predicted reduction in task-dependent map entropy. Under the field-of-view-local approximation, only visible cells contribute to that reduction.

F. Expected Information Gain and NBV Selection

VI. HIERARCHICAL-TASK MPC

HTMPC gives base-trajectory tracking priority over the hand-camera viewpoint task while enforcing environment and self-collision constraints. Its two-stage implementation preserves the first-stage base-tracking cost while optimizing camera motion.

VI. HIERARCHICAL-TASK MPC

  • The mobile base and arm are modeled through generalized coordinates and velocities, with accelerations as control inputs.For a holonomic base, the model reduces to a double integrator.Source: §VI · p. 10
  • HTMPC prioritizes the base pose-tracking objective over the hand-camera objective, using remaining degrees of freedom to support active perception.The desired base pose comes from the planned navigation path, while the camera pose comes from the selected NBV.Source: §VI · pp. 10–11
  • Environment collision avoidance uses CBF clearance-rate constraints, while self-collision avoidance constrains distances between selected nonadjacent collision-sphere pairs.The authors choose CBFs for environment avoidance because online maps are affected by perception delays, occlusions, and uncertainty.Source: §VI · pp. 10–11
  • The controller uses two consecutive regularized STMPCs: the second optimizes the camera task subject to a bound on the first-stage base-tracking cost.The receding-horizon controller uses the latest robot state, maintained map, and base and camera references.Source: §VI · p. 11

VII. EXPERIMENTS

The experiments use an object-mapping and localization stack with a path-guided discrete viewpoint set and fixed controller parameters. Baselines vary how the hand-camera pose is chosen, separating passive, path-directed, and uncertainty-only sensing from the full method.

1) Platform and System Configuration:

  • Modified ORB-SLAM3 estimates robot state from the base camera and wheel odometry at 50 Hz; dense object mapping integrates base- and hand-camera observations at 5–10 Hz in parallel.Source: 1) Platform and System Configuration · p. 11
  • The implementation samples hand-camera viewpoints using five yaw offsets and two reaches around a look-ahead waypoint, shortening reaches that approach obstacles.HTMPC runs at 10 Hz with a 3 s horizon; final commands are sent at 100 Hz, and parameters are fixed across experiments.Source: 1) Platform and System Configuration · p. 12

2) Baseline Methods:

  • The four baselines vary hand-camera pose selection, from fixed or random sensing to path lookahead and uncertainty-only NBV without task weighting.The full method, NBV-Map-Task, combines map uncertainty with task relevance; NBV-Map omits the task-weighting term.Source: 2) Baseline Methods · p. 12

3) Experiment Protocol:

The experiments progress from static-scene viewpoint selection to controlled object shifts and repeated real-world visits. In the static benchmark, the proposed method recovers hidden occupied regions in hard scenes, though increased coverage trades off against precision and false-positive rate.

3) Experiment Protocol:

  • The benchmark evaluates three scenes over three consecutive loops per scene, carrying forward and updating the map after successful loops and resetting after collisions.The static benchmark isolates viewpoint selection under occlusion; the other experiment families test controlled object shifts and repeated real-world visits after a shelf relocation.Source: 3) Experiment Protocol · pp. 12–13

1) Scene Overview and Evaluation Metrics:

  • The scenes contain blind corners, narrow passages, and path-occluded obstacles; the robot plans online from blue waypoints and must observe hazards before entering turns.The benchmark includes open loops as well as sharper turns and stronger occlusions; the proposed method directs its hand camera toward hidden hazards before the maneuver.Source: 1) Scene Overview and Evaluation Metrics · p. 13
  • The static-scene evaluation measures voxel-map precision, recall, and false-positive rate, task-dependent entropy along executed paths, and navigation success and clearance.Maps are assessed against manually constructed reference maps; reconstruction is evaluated after the final loop, while entropy, success, and clearance include all loop executions.Source: 1) Scene Overview and Evaluation Metrics · pp. 13–14

2) Map Reconstruction Quality:

  • In Scene 2, the proposed method recovers occupied voxels behind the L-shaped blind corner and is the only method that completes the scene.In Scene 3 it captures more cluttered occupied voxels supporting safe turns; in Scene 1 all methods complete and it has the best precision, recall, and FPR.Source: 2) Map Reconstruction Quality · p. 14
  • Combining map information with task relevance increases hidden-obstacle coverage, but in hard scenes this comes with lower precision and higher FPR.Fixed and Random do not reliably inspect hidden obstacles before turns; Lookahead helps when path direction exposes them, while NBV-Map often selects outward frontiers.Source: 2) Map Reconstruction Quality · p. 14

3) Task-Dependent Map Entropy:

The proposed framework reduces task-relevant uncertainty and navigation risk by directing sensing toward uncertain regions that matter to the current maneuver. Controlled scene changes show that sensing responds to displaced objects, while real-world visits show map updates guiding replanning.

3) Task-Dependent Map Entropy:

  • Across Scenes 2 and 3, task weighting directs observations toward uncertainty near the upcoming turn, where several baselines collide before completing a loop.In Scene 1, all methods complete the task, but the proposed method achieves lower mean and variance in task-dependent map entropy.Source: 3) Task-Dependent Map Entropy · p. 14

4) Navigation Safety:

  • The proposed method maintains more consistent clearance in harder scenes, while Lookahead still has many near-collision events in Scene 3 despite completing loops.In Scene 1, methods are more comparable, although several baselines approach or cross the zero-clearance boundary.Source: 4) Navigation Safety · p. 15

C. Controlled Simulation Under Scene Changes

  • After boxes shift, hand-camera headings bias toward the changed side even though the base trajectory remains similar across loops.The controlled experiment keeps the waypoint sequence and environment otherwise unchanged, isolating object-level scene changes; object and free-space confidence decay is disabled.Source: C. Controlled Simulation Under Scene… · pp. 15–16

D. Real-World Long-Term Scene Changes

  • Across four real-world visits with the same waypoint sequence, the robot inspects the relocated shelf and switches between outside and inner routes according to its position.New observations reveal map inconsistency, raise entropy near the shelf, attract further sensing, and precede replanning.Source: D. Real-World Long-Term Scene Changes · p. 17

VIII. CONCLUSIONS

  • Real-world experiments show reduced task-relevant map uncertainty, improved map consistency, and lower navigation risk than passive or non-task-aware baselines.The static navigation benchmark also reports that the proposed method alone completes all tested loops; each baseline fails in at least one harder scene.Source: 4) Navigation Safety · p. 15, §VIII · p. 17

Related papers

Loading 2610.29964v1…