Source-linked AI summary
Macro-Action Topological Navigation under Noisy Localization using Reinforcement Learning
Simon Hakenes, Tobias Glasmachers
TL;DR
Large photorealistic navigation from pixels is difficult when self-localization is unavailable, yet the paper tests whether global metric accuracy is necessary. It replaces the pose oracle with object-centric visual localization and finds that locally consistent drift can still support target-reaching, within simulator and detection limits.
Problem
The problem is whether macro-action navigation can work without ground-truth pose when localization is noisy, rather than relying on a globally consistent metric map.
Method
The method combines object-centric ORB landmarks, pose measurements, an EKF motion model, map-write gating, and macro-action control over a topological map.
Results
The Full system reaches target objects despite global position drift of several meters, while locally consistent pose remains sufficient for the controller.
Takeaways & Limitations
The motion model keeps heading usable between measurements, while Kabsch measurements prevent position from drifting away.
Takeaways & Limitations
The evaluation still relies on simulator-provided object masks and ids, and the hand-built controller carries much of navigation reliability.
Abstract
from arXiv · showhide
Navigating large, photorealistic 3D apartments from raw pixels is widely considered infeasible for plain reinforcement learning. We build an agent that does it anyway, estimating its own pose from the camera alone. The agent has to reach several target objects in sequence, and their positions change between episodes, so it must explore to find them. It builds on our earlier object-centric topological controller, which still read the agent's true pose and its object detections from the simulator. Here we replace that true pose with an onboard, object-centric estimate. For each object we keep a bank of ORB features that, when the object is seen again, yield a rough pose measurement, which a minimal Extended Kalman Filter (EKF) fuses with a motion model. As on a real robot, the executed motions are noisy. The estimate drifts, but the agent and the nearby objects drift together, so a locally consistent pose is enough to follow each short edge and then home in visually on the target, which lets us replace full SLAM with a much smaller model, closer to how biological navigation appears to work. In the photorealistic Habitat simulator, the agent reaches its target objects from vision alone, with a pose that only needs to be locally consistent.
1 Introduction
The paper argues that navigation can rely on a globally inaccurate but locally consistent topological pose, replacing ground-truth localization with a camera-based object-centric estimate. In Habitat, this supports sequential object navigation despite noisy motion and pose drift.
- Topological maps represent landmarks and traversable links without requiring globally exact or globally consistent metric reconstruction.
- The earlier DQN agent made large-scene pixel navigation tractable by selecting object-centric macro actions over a growing topological map.
- The approach tests whether locally consistent noisy pose is sufficient for macro-action navigation instead of requiring full global SLAM.
- The new system removes simulator pose access and estimates camera-based pose using per-object ORB features, rough re-observation measurements, and an EKF motion model.
- No loop closure or global optimization is used; drift remains acceptable because nearby objects and the agent drift together across short macro-action edges.
2 Related Work
Related work spans globally consistent SLAM, learned metric localization, topological navigation, and macro-action control. This paper combines a small classical localization stack with a DQN that chooses which object to visit.
- Classical SLAM uses feature tracking, loop closure, and bundle adjustment to build globally consistent metric reconstructions.
- Learned navigation systems estimate pose or build metric maps from visual streams, often combining RGB or RGB-D inputs with learned SLAM-like components.
- Topological navigation methods store views or learn graph construction and localization, whereas this paper uses a small classical stack for localization.
- Its architecture separates an inherited object-centric DQN from a new navigation and localization layer that replaces the ground-truth pose oracle.
- The system retains object-centric macro actions that reduce navigation to high-level object selection rather than reward-trained elementary action sequences.
3 Method
The method separates semantic object selection from geometric execution: a DQN chooses known objects, while a map-based controller plans and executes motion using an estimated pose.
- The system has a decision layer for object-centric macro-action selection and a navigation-localization layer for shortest-path motion execution.
- The new localizer replaces simulator pose with ORB landmark matching, aligned pose measurements, EKF fusion, and an elementary-action motion model.
- A macro action navigates to a specific object stored in the topological map and ends when the controller declares that object Found.
- At time t, the available macro actions are the known object nodes, At = Ot, so the action space grows with the map.
- Each object node stores image patches, while task progress is represented by a one-hot vector identifying the active target-sequence step.
- A shared network evaluates each known object separately, producing one Q-value per candidate instead of requiring a fixed output neuron for every action.
3.2 DQN architecture and policy
The DQN represents each candidate object with image patches and evaluates candidates individually alongside task progress. Policy decisions are then made from the resulting object-wise Q-values.
- Up to Ni image patches represent each object, and all patches share the same convolutional feature extractor.
- The extracted object features are combined with the task-progress vector through an outer product before a fully connected layer outputs one Q-value.
- At decision time, the network computes one Q-value for every known object and uses mostly greedy selection with Boltzmann exploration.
3.3 Map representation
The map is an object-centric topological graph whose semantic object nodes support macro-action selection, while backbone nodes provide safe navigation waypoints. It grows online from visual observations and tolerates global coordinate drift by relying on local geometry.
- Map structure: The map is an undirected weighted graph containing backbone nodes for visited poses and object nodes for semantic landmarks.Only object nodes are offered to the DQN as macro actions; backbone nodes remain internal navigation structure.
- Navigation roles: Object nodes are poor direct waypoints because objects may be unreachable, so the controller routes to a nearby backbone node before visual homing.
- Online growth: The map starts empty and expands during exploration, with estimated poses added as backbone nodes while navigation proceeds on the partial graph.
- Object observations: Visible objects are added from RGB-D images and simulator-provided semantic masks, with patches and depth-derived 3D positions stored for each object.Object detection itself is treated as a separate problem and is not addressed here.
- Local consistency: The map is not globally consistent, but relative node-to-node geometry remains usable even when absolute coordinates drift by meters.
- Visual correspondence: ORB features are stored per object to provide stable correspondences when that object is seen again.New keypoints extend an object’s feature bank but are not used as localization correspondences in their first frame.
3.5 Kabsch pose measurement
The Kabsch-based localizer estimates a planar rigid transform from matched object-feature anchors. Weighted correspondences produce a translation and rotation measurement that the EKF can use to correct pose drift.
- Correspondences: Matched ORB features provide ground-plane point pairs linking current agent-frame observations to stored map anchors.
- Weighting: Inverse-distance weights reduce the influence of far, noisier anchors in the rigid alignment.
- Rigid alignment: The weighted cross-covariance SVD determines the planar rotation aligning observed points with their stored anchors.
- Pose output: The determinant correction selects a proper rotation rather than a reflection, while the translation and yaw form the measured position and heading.That combined pose measurement is passed to the EKF.
- Measurement confidence: Residuals ri = ∥R pi + t −qi∥ quantify match agreement and provide a measure of confidence in the pose measurement.
3.6 Extended Kalman filter
The EKF combines commanded-motion prediction with intermittent visual pose measurements to maintain an SE(2) estimate under noisy actuation. It rejects inconsistent corrections and continues predicting when visual features are unavailable or unreliable.
- Filter structure: The EKF fuses elementary-action geometry with Kabsch measurements into a pose estimate updated at every step.
- State: The pose state uses ground-plane coordinates x and z together with yaw ψ.
- Motion model: A forward action of length d updates position according to the current yaw while leaving yaw unchanged.The prediction follows the commanded elementary action.
- Motion model: TurnLeft and TurnRight actions keep position fixed and change yaw by the known turn angle.
- Noisy execution: Actuation noise uses standard deviations of 1° for turning, 0.01 m for forward steps, and 0.005 m for sideways slip, causing believed-pose drift.
- Robust updates: The filter coasts on the motion model when no useful features are matched and accepts visual corrections only when they fit the current belief.
3.7 Map-write gating
Because incorrect pose estimates can corrupt the map and mislead later navigation, the system gates map writes using localization confidence and postpones uncertain observations.
- Failure mode: A wrong pose can place an object incorrectly on the map, causing that landmark to mislead later planning and navigation.
- Confidence gating: The system writes observations only when feature matches and EKF certainty indicate that the current pose is trustworthy.
- Deferred updates: Uncertain observations are deferred so objects are added late rather than stored at incorrect positions.
3.8 Macro-action execution under noisy localization
The controller follows graph-based routes through short waypoint steps, then switches to visual homing near the target. This keeps execution tied to the live image rather than requiring globally accurate pose estimates.
- Macro-action execution: The controller plans the shortest graph path and follows it one waypoint at a time toward the selected object.It turns toward each waypoint and moves forward, using fallback behaviors to remain reliable.
- Visual homing: Near the target, visual homing turns toward the object when it is off-center and moves forward when it is centered.The target is identified from its semantic mask and depth; reaching sufficient proximity triggers Found.
- Visual homing: Visual homing limits the effect of pose drift because the live camera image guides the final approach and reanchors the estimate when the object is seen.The controller therefore needs accurate localization least at the point where it must reach the target.
4 Experiments and Results
The experiments test whether object-centric macro-action navigation remains effective when ground-truth pose is replaced by noisy visual localization. In Habitat, the full localizer is compared with ablations, an oracle, and a random macro policy using genuine success, action cost, and localization error.
- Experimental setup: The Habitat task requires finding a fixed sequence of target objects from an empty map while exploring and building the topological map online.An episode succeeds only when every target in the sequence is reached in order.
- Experimental setup: The central experiment replaces simulator ground-truth pose with noisy localization while keeping object masks and identities supplied by the simulator.This isolates the localization stack as the new element under test.
- Experimental setup: Five conditions compare the complete ORB-Kabsch-EKF system, two localization ablations, a ground-truth-pose oracle, and random macro selection.No-EKF removes filtering and the motion model, Motion-Only removes visual measurements, GT-Pose removes actuation noise, and Random-Macro bypasses the trained DQN.
- Results: 84%, 62%, and 46% are Random-Macro’s conventional target success rates for one, two, and three targets, respectively.Conventional success includes Found declarations that may result from give-ups landing inside the success radius.
- Results: The Full system’s genuine single-target success falls from 93% to 85%, while Random-Macro falls from 84% to 54%.The genuine metric counts only episodes where the agent actually reached and stopped at the target.
- Results: Three targets under terminal reward is the hardest condition, yet Full completes most sequences and has the highest genuine success among noisy conditions.Terminal reward reinforces the sequence only after completion, creating the hardest credit-assignment setting.
- Results: Full maintains 13–39 degrees of heading error and position drift of a few meters, while No-EKF reaches 80–104 degrees and Motion-Only drifts 3–4 meters.GT-Pose achieves the highest genuine success overall; among noisy-pose conditions, Full is strongest.
5 Discussion and Conclusion
The experiments show that macro-action topological navigation can operate with a globally drifting but locally consistent pose, though it remains less reliable than ground-truth localization. The method still has important practical limitations, including costly ORB features, assumed perfect object detection, and reliance on hand-built low-level control.
- Local consistency is sufficient for the controller to keep working despite global position drift of several meters, but performance remains below the ground-truth oracle.The gap widens as target sequences grow longer, so localization noise is not harmless.
- The Full system succeeds because nearby objects drift jointly with the agent, preserving the relative geometry needed for short macro-action edges and visual homing.Visual homing completes target approach from the image once the target is visible.
- The No-EKF ablation reaches heading errors of 80–104 degrees, showing that the explicit motion model is necessary for usable pose estimates.Its low action counts reflect easy episodes it wins rather than robust navigation.
- The Motion-Only ablation develops several meters of position error and 18–43 degrees of heading error under noisy actuation because dead reckoning receives no measurement correction.Together, the ablations assign heading stabilization to the motion model and position-drift correction to visual measurements.
- The system could potentially replace depth with cheaper monocular scale sources because its localizer requires local rather than globally accurate depth.Suggested alternatives include ORB-anchor triangulation, learned monocular depth, and object-mask apparent size.
- Three practical limitations remain: ORB feature extraction is costly, object masks and IDs come from the simulator, and low-level execution relies heavily on hand-built control.The authors identify learned features, real object detection, and learned low-level execution as future directions.