Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors

Rico Jonschkowski; Divyam Rastogi; Oliver Brock

Source-linked AI summary

Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors

Rico Jonschkowski, Divyam Rastogi, Oliver Brock

arXiv:1805.11122v2May 28, 2018cs.LGcs.AIcs.ROstat.ML

TL;DR

State estimation needs models that learn effectively while preserving the recursive structure of uncertain inference. The paper introduces differentiable particle filters with learnable motion and measurement models, reporting lower error, improved data efficiency and generalization, and a limitation from nondifferentiable resampling.

Problem
End-to-end learning can optimize system performance, but robotic state estimation also requires priors that encode its structured filtering problem and regularize learning.
Method
DPFs make the particle-filter algorithm differentiable and learn its motion and measurement models end-to-end within an architecture that encodes the Bayes-filter prior.
Results
Compared with LSTMs, DPFs reduce error rates by ∼80% or require 87% less training data for the same error rate, while remaining robust to policy changes.
Takeaways & Limitations
Algorithmic priors can improve state-estimation performance, explainability, data efficiency, and generalization while retaining end-to-end learning.
Takeaways & Limitations
Nondifferentiable resampling stops gradient computation after one loop iteration, limiting the implementation and complicating backpropagation through multiple time steps.

Abstract

from arXiv · show

We present differentiable particle filters (DPFs): a differentiable implementation of the particle filter algorithm with learnable motion and measurement models. Since DPFs are end-to-end differentiable, we can efficiently train their models by optimizing end-to-end state estimation performance, rather than proxy objectives such as model accuracy. DPFs encode the structure of recursive state estimation with prediction and measurement update that operate on a probability distribution over states. This structure represents an algorithmic prior that improves learning performance in state estimation problems while enabling explainability of the learned model. Our experiments on simulated and real data show substantial benefits from end-to- end learning with algorithmic priors, e.g. reducing error rates by ~80%. Our experiments also show that, unlike long short-term memory networks, DPFs learn localization in a policy-agnostic way and thus greatly improve generalization. Source code is available at https://github.com/tu-rbo/differentiable-particle-filters .

I. INTRODUCTION

The paper turns robotic algorithms into differentiable network architectures, using algorithmic priors to combine end-to-end learning with structure for state estimation. Differentiable particle filters improve performance, data efficiency, explainability, and policy generalization relative to generic learned models.

Motivation: End-to-end learning optimizes the complete system for performance rather than separately optimizing each component.This is useful when suitable objectives for individual components are unknown.
Algorithmic priors: Algorithmic priors encode robotic problem structure in differentiable architectures, restricting the hypothesis space while allowing components to adapt jointly.This balances end-to-end flexibility with regularization against overfitting.
Differentiable particle filters: DPFs implement particle-filter state estimation with learnable motion and measurement models, recursively updating a probability distribution over states.They apply prediction and measurement updates within an end-to-end differentiable architecture.
Results: Compared with LSTMs, DPFs reduce error rates by ∼80% or require 87% less training data for the same error rate.DPFs also remain explainable because their learned models and interactions can be examined.
Results: DPFs generalize across policies, whereas LSTMs fail when tested with a different policy than used during training.The particle-filter structure supports policy-agnostic localization in the reported experiments.
Related work: Sample-based differentiable representations can model multimodal distributions and focus computation on probable states, but require solutions for sampling, density estimation, and nondifferentiable resampling.The paper positions DPFs as extending differentiable belief representations beyond histogram and Gaussian approaches.

III. BACKGROUND: BAYES FILTERS AND THEIR PARTICLE-BASED APPROXIMATION

Bayes filters estimate a latent state by recursively updating a belief distribution from actions and observations. Their prediction and measurement steps use motion and measurement models, while continuous-state implementations approximate the belief in different ways.

State estimation: State estimation infers a robot’s latent state from observation and action histories when individual observations may be insufficient.The estimate is represented as a belief distribution over the current state.
Bayes filters: Under the stated factorization assumptions, the Bayes filter is optimal and recursively updates the current belief from the previous belief, action, and observation.The Markov property makes the previous belief sufficient for predicting the future.
Bayes-filter steps: The prediction step uses the motion model p(s_t | s_t−1, a_t) to compute a predicted belief from possible previous states.It sums over states from which the action could have led to the current state.
Bayes-filter steps: The measurement update uses p(o_t | s_t) to weight states according to the likelihood of the current observation and normalizes the resulting belief.This update applies Bayes’ rule.
Approximations: Continuous-state Bayes filters approximate beliefs using representations such as histograms, Gaussians, or particle sets.Each representation provides a different implementation of the same filtering structure.

B. Particle Filters

Particle filters approximate a belief distribution with weighted samples. They implement prediction by moving particles, measurement update by changing weights, and resampling by concentrating particles on probable states.

Representation: Particle filters represent the belief with a set of particles and associated weights.The particle set approximates the continuous state distribution.
Filter update: Each iteration moves particles, changes their weights, and resamples them proportionally to weight.Resampling duplicates high-weight particles and removes low-weight particles.
Filter update: Resampling makes particle filtering efficient by focusing the belief approximation on probable states.Computational effort is concentrated where the estimated probability is higher.
Filter update: The prediction step moves each particle stochastically by sampling from a generative motion model.The measurement update assigns each particle a weight equal to the observation likelihood.
Filter update: The particle set is resampled by randomly drawing particles in proportion to their weights before the next prediction and update iteration.This completes the recursive particle-filter cycle.

IV. DIFFERENTIABLE PARTICLE FILTERS

Differentiable particle filters embed particle filtering in a recurrent, end-to-end differentiable architecture with learnable models. Their prediction step samples noisy actions and applies dynamics, while the particle belief is updated recursively from actions and observations.

Architecture: DPFs are differentiable particle filters with end-to-end learnable models and a recurrent architecture encoding the particle-filter algorithmic prior.They represent a belief over states using the structure of particle filtering.
Architecture: End-to-end learning optimizes the models for the filter’s output by backpropagating through the algorithm rather than changing the algorithm itself.The objective is state-estimation performance.
Implementation: The implementation is publicly available as TensorFlow and Sonnet source code.The paper provides a repository for the DPF implementation.
Architecture: DPFs update a weighted-particle belief recursively using the previous belief, action, and observation.The belief contains particle states and weights at each time step.
Prediction: The prediction step samples a probabilistic motion model for each particle, then moves particles using the resulting noisy actions and dynamics model.The action sampler and dynamics model can be learned, while known dynamics can also be implemented directly.

C. Measurement Update

The DPF measurement model uses observations both to update particle weights and to propose new particles. It combines learned observation encoding, particle proposal, likelihood estimation, and density-based belief evaluation.

Measurement model: The measurement model encodes each observation, proposes particles, and estimates each particle’s observation likelihood.The three components are the shared encoder h, proposer k, and likelihood estimator l.
Measurement update: DPFs use observations to propose new particles in addition to computing particle weights.
Particle proposal: Dropout supplies randomness that lets the proposer generate different particles from the same observation encoding.It is used as a sampling source rather than as regularization.
Particle proposal and resampling: During filtering, DPFs gradually shift from proposed hypotheses toward resampling, which tracks and removes hypotheses.The proposed-to-resampled particle ratio follows γ^(t−1), with γ set to 0.7 in the experiments.
Learning objective: The models can be learned from supervised sequences by maximizing belief at the true state.Particle weights are treated as a Gaussian mixture for estimating belief, with state dimensions scaled by average step size.

1) Individual learning of the motion model:

Individual motion-model learning trains the action sampler and dynamics components against observed motion and state transitions. The procedure uses likelihood maximization for known motion noise and mean squared error when learning unknown dynamics.

Motion model: The action sampler is optimized to match observed motion noise by maximizing the likelihood of sampled states and actions.
Dynamics model: When the dynamics model is unknown, its learnable network is trained by minimizing mean squared error against observed state transitions.
Observation likelihood: The observation likelihood estimator is trained to assign high likelihood to observations in their true states and low likelihood elsewhere.

3) End-to-end learning:

DPFs are trained end-to-end through filtering, but nondifferentiable resampling limits gradients to a single iteration. Experiments evaluate this approach across simulated localization and real visual odometry, showing strong gains over relevant baselines.

End-to-end learning: DPFs compute end-to-end gradients by backpropagating from the output through the filtering loop, with training applied to overlapping subsequences.The objective maximizes belief at the true states along each sequence.
Limitations: Resampling stops gradient computation after one loop iteration, so gradients omit how earlier prediction and update steps affect the current belief.This limitation constrains the current implementation.
Experiments: The evaluation covers global localization in simulated mazes and visual odometry on KITTI, using known dynamics while learning measurement models from data.The tasks test both multiple-hypothesis state estimation and applicability to real visual data.
Results: ∼80% error reduction is reported for DPF algorithmic priors compared with LSTMs, while localization remains useful under different policies.The reported policy-shift comparison gives ∼84% error for the LSTM and ∼15% for the DPF.
Results: DPFs outperform BKFs on visual odometry despite the task matching Kalman-filter assumptions, and they learn measurement models from less than 40 minutes of video.The result is reported on real data with diverse observations and task properties differing from global localization.

A. Global Localization Task

DPFs perform global localization by combining learned motion and measurement models within an explainable particle-filtering process. The learned models adapt uncertainty and particle weights so hypotheses converge toward the true state.

Task: The global localization task estimates robot pose from visual and odometry input in partially observable simulated mazes.The mazes remove unique wall textures and objects, requiring the system to reason from incomplete observations.
Explainability: DPFs remain explainable after end-to-end learning because their motion model, measurement model, and filtering interactions can be inspected.Models were generally learned individually before end-to-end refinement.
Motion model: The learned motion model spreads particles according to movement magnitude, assigning greater uncertainty to larger steps.Individual learning matches odometry noise, whereas end-to-end learning overestimates noise in ways that can support filtering.
Measurement model: The measurement model proposes particles and assigns high weights to the true state and locally symmetric maze states.Its learned distribution also reflects the hand-coded policy, such as emphasizing dead ends where the robot looks before turning.
Filtering: Filtering begins with many hypotheses, forms clusters, removes observation-inconsistent clusters, and ultimately tracks the true state.The weighted particle mean matches the true state after the first few steps in the illustrated trajectory.

2) End-to-end learning improves performance:

End-to-end learning improves DPF state-estimation performance, especially when preceded by individual model training. Compared with an LSTM baseline, DPFs achieve lower error rates and reach the LSTM’s final performance with less data.

Learning performance: End-to-end-trained DPFs consistently outperform individually trained DPFs across all mazes and training-set sizes.Individual training performs worst with fewer than 64 trajectories and plateaus beyond 125 trajectories because its models are not optimized for state-estimation performance.
Learning performance: Sequencing individual and end-to-end learning improves performance further by combining model-specific information with end-to-end performance information.The individual stage provides additional information about each model, while end-to-end learning adapts their interaction for filtering.
LSTM comparison: DPFs achieve lower error rates than the LSTM baseline for every maze and training-data amount.The comparison uses a generic two-layer LSTM with the same convolutional image encoder and end-to-end mean-squared-error training.
Data efficiency: DPFs reach the LSTM’s final performance with 125 trajectories, using 1/8 of the full training set.With a learned dynamics model, DPF error rises from 1.6% to 2.7%, still below the LSTM’s 6.0%.

4) Algorithmic priors lead to policy invariance:

DPFs generalize across changes in the robot’s data-collection policy, whereas LSTMs can overfit policy-specific state-action correlations. Training on both policies also improves cross-policy performance for all methods.

Policy invariance: When training and test policies differ, LSTM error rises above 80%, while DPF error remains 5% or 26% in the reported settings.All methods have low error when tested on their training policy.
Policy invariance: LSTMs fail to generalize because they can infer state from action-policy correlations rather than separating action and observation information.For example, movement-speed differences between hallways and rooms can become shortcuts for state estimation.
Policy invariance: DPFs cannot directly infer states from actions, helping prevent the policy-specific shortcut available to the LSTM baseline.This structural distinction is the paper’s stated explanation for the stronger policy generalization of DPFs.
Policy invariance: DPFs generalize better from policy A to policy B than in the reverse direction because states visited by A cover those visited by B, but not vice versa.The asymmetry is not attributed to end-to-end overfitting because it also appears with individually learned models.
Policy invariance: Combining training data from policies A and B yields good test performance under either policy for all methods.This domain-randomization-style strategy helps by covering the union of visited states and supplying both policies’ state-action correlations.

B. Visual Odometry Task

DPFs also work for visual odometry on real KITTI data, where they outperform backpropagation Kalman filters despite the task’s unimodal-belief setting. The result supports applying differentiable particle filtering across different state-estimation conditions.

Dataset and task: The visual odometry evaluation uses eleven real-car KITTI trajectories totaling 40 minutes of RGB stereo video with ground-truth pose.The data span urban driving and test generalization across visually diverse observations.
Model: The DPF uses a five-dimensional state containing position, orientation, forward velocity, and angular velocity.It starts from a known initial state and uses a first-order dynamics model with a learnable action sampler.
Results: DPFs outperform BKFs, reducing error by approximately 30% for short sequences.Evaluation uses eleven-fold cross-validation and averages errors over subsequences of 100, 200, 400, and 800 time steps.
Results: DPFs outperform BKFs even though visual odometry fits Kalman-filter assumptions of a unimodal belief and known initial state.The authors suggest particle representations may help capture long-tailed probability distributions.
Implications: The visual odometry result demonstrates that DPFs generalize to a different task and can operate successfully on real data.The conclusion frames end-to-end learning with algorithmic priors as improving data efficiency and generalization.