Source-linked AI summary
Particle Filter Networks with Application to Visual Localization
Peter Karkus, David Hsu, Wee Sun Lee
TL;DR
Particle filtering requires probabilistic system models that are difficult to construct for complex dynamics and rich visual observations. PF-net embeds learned transition and observation models with the particle-filter algorithm in a differentiable neural network trained end-to-end, and it outperforms alternative architectures and a conventional model-based method while generalizing to unseen environments.
Problem
Constructing probabilistic system models for particle filtering is difficult when dynamics or sensory inputs, such as camera images, are complex.
Method
PF-net is a differentiable recurrent neural network that jointly encodes learnable transition and observation models with the particle-filter algorithm and trains them end-to-end.
Results
PF-net consistently outperformed alternative learning architectures and a conventional model-based method across visual-localization settings, including unseen environments.
Takeaways & Limitations
End-to-end training lets PF-net learn models optimized for particle filtering and localize successfully in challenging unseen environments.
Takeaways & Limitations
With RGB input, PF-net requires approximately 0.6 ms per particle per step, and high-uncertainty indoor localization may require 1,000–10,000 particles.
Abstract
from arXiv · showhide
Particle filtering is a powerful approach to sequential state estimation and finds application in many domains, including robot localization, object tracking, etc. To apply particle filtering in practice, a critical challenge is to construct probabilistic system models, especially for systems with complex dynamics or rich sensory inputs such as camera images. This paper introduces the Particle Filter Network (PFnet), which encodes both a system model and a particle filter algorithm in a single neural network. The PF-net is fully differentiable and trained end-to-end from data. Instead of learning a generic system model, it learns a model optimized for the particle filter algorithm. We apply the PF-net to a visual localization task, in which a robot must localize in a rich 3-D world, using only a schematic 2-D floor map. In simulation experiments, PF-net consistently outperforms alternative learning architectures, as well as a traditional model-based method, under a variety of sensor inputs. Further, PF-net generalizes well to new, unseen environments.
1 Introduction
The paper introduces PF-net, which embeds a particle-filter algorithm and learned probabilistic models in one differentiable network trained end-to-end. It targets visual localization where a robot matches rich 3-D observations to a schematic 2-D map and generalizes to unseen environments.
- 1 Introduction: PF-net jointly encodes learnable transition and observation models with the particle-filter algorithm in a single recurrent neural network.The unified representation trains models optimized for particle filtering rather than generic system models.
- 1 Introduction: Visual localization requires matching rich 3-D camera features to crude 2-D map geometry while ignoring unmapped objects such as furniture.The observation space is enormous, making conventional image-conditioned probabilistic models difficult to construct.
- 1 Introduction: Simulation experiments show PF-net localizes effectively in new, unseen furnished environments and outperforms a conventional model-based method.The paper also reports sensor fusion across RGB and depth cameras and integration of semantic map information.
2 Background
Particle filters represent state beliefs with weighted particles and update them through stochastic transitions, observation likelihoods, normalization, and resampling. Unlike Gaussian or discrete histogram assumptions, particle sets can represent arbitrary continuous, multimodal, non-Gaussian distributions.
- 2 Background: Differentiable algorithm priors extend algorithm-embedded learning to state estimation, including Kalman and histogram filter networks.Kalman filtering assumes unimodal Gaussian beliefs, whereas histogram filtering uses discrete states and faces scaling difficulties in high-dimensional spaces.
- 2 Background: Particle filters approximate the posterior belief over states with weighted particles, supporting arbitrary continuous, multimodal, non-Gaussian distributions.The number of particles is K, while s_k and w_k denote particle states and weights.
- 2 Background: Each update samples particle states from a transition model conditioned on the previous state and action, then updates weights using observation likelihoods conditioned on state and map.The transition model is T, while the observation model Z defines the conditional observation probability.
- 2 Background: Resampling addresses particle degeneracy by selecting particles in proportion to their weights and resetting the new weights uniformly.The resulting set concentrates representation on important belief regions, although repeated particles may occur.
3 Particle Filter Network
PF-net combines learnable transition and observation models with a differentiable particle-filter algorithm in one recurrent network. Its task-specific end-to-end training lets the learned model focus on features useful to inference rather than represent a generic probability distribution.
- PF-net encodes particle filtering, including learnable transition and observation models, in a single recurrent neural network with shared weights across particles.The remaining computation graph implements particle-filter operations, which must remain differentiable for backpropagation.
- The network’s algorithm-specific structure provides a more efficient learning prior than generic recurrent architectures such as LSTM.
- End-to-end optimization through inference tailors the learned model to a specific particle-filter algorithm and task.This can reduce the need to model complex conditional distributions by learning only task-relevant features.
- PF-net makes particle filtering differentiable by using reparameterization for sampling and soft-resampling to preserve gradients through particle updates.Soft-resampling trades off the desired sampling distribution against a uniform distribution; carrying particles forward can work better in low-uncertainty training, while soft-resampling works better under high uncertainty.
- The architecture can use different particle counts during training and evaluation, while task-specific state representations and transition and observation networks must be selected.
4 Visual localization
The visual-localization application estimates a robot’s pose from camera observations, odometry, and a schematic 2-D map in an unseen indoor environment. PF-net addresses multimodal uncertainty and difficult map-image matching with discriminative learned likelihoods and a structured observation network.
- PF-net addresses visual localization where ambiguous observations require maintaining a multimodal, non-Gaussian continuous posterior over robot states.The robot navigates an unseen indoor environment using a camera, odometry, and a schematic 2-D floor map.
- Its observation model learns particle likelihoods directly from a particle pose, camera image, and floor map instead of modeling the full conditional distribution of images.This discriminative model is trained end-to-end without supervision on particle likelihoods.
- The visual task is difficult because rich 3-D camera features must be matched to crude 2-D map geometry while handling objects such as furniture that are absent from the map.
- The observation architecture compares map geometry with visual features by transforming the global map into a particle-specific local view using a Spatial Transformer Network.Separate learned CNN components extract local-map and camera-image features before combining them to produce a particle likelihood.
- The model represents each particle as a candidate pose and weight, propagates pose estimates using odometry in the global frame with Gaussian noise, and outputs their weighted mean.The pose is st = {x_t, y_t, φ_t}; σ_t and σ_r are manually chosen translation and rotation noise standard deviations in the experiments.
5 Simulation experiments
Simulation experiments evaluate PF-net for visual localization across environments, uncertainty levels, sensors, semantic maps, and alternative learning architectures. PF-net localizes successfully under high initial uncertainty and outperforms comparison methods while benefiting from end-to-end sensor and map processing.
- Main results: PF-net successfully localized without additional training when the initial belief was uniform over one room and even over the entire floor map.Evaluation covered tracking, semi-global localization, and global localization on previously unseen buildings.
- Main results: PF-net consistently outperformed the histogram-filter and LSTM networks, combining particle-based belief representation with Bayesian updates while avoiding their discrete-state or unstructured-belief limitations.The histogram-filter network is restricted to discrete beliefs, whereas the LSTM relies on a hidden state vector without structure specific to probabilistic state estimation.
- Main results: PF-net performed significantly better than a known LIDAR model when environments contained objects absent from the map, while both methods performed similarly in furniture-free settings.The authors attribute this robustness to end-to-end learning of relationships between beams and accommodation of map imperfections.
- Main results: PF-net with RGB images was almost as effective as with depth images, outperformed simulated LIDAR, and improved further when RGB and depth inputs were combined.These results indicate learned geometry extraction from RGB images and end-to-end sensor fusion.
- Main results: Simple semantic maps containing door and room-category labels improved localization performance.The semantic labels were encoded in separate input channels and evaluated for semi-global localization.
6 Conclusion & future work
PF-net combines learned probabilistic models with a differentiable particle-filter algorithm and successfully localizes in challenging unseen environments. Future work targets broader applications, real-world deployment, robustness, and online execution efficiency.
- PF-net combines learned probabilistic models with a particle-filter algorithm in a differentiable network for sequential state estimation.
- Through end-to-end training, PF-net localizes successfully in previously unseen environments containing objects absent from the map.
- Real-world deployment remains future work, with concerns about online execution speed and the particle counts required under high uncertainty.
- Future work includes applying PF-net to visual tracking and SLAM, learning latent state representations, and supporting planning under partial observability.