Source-linked AI summary
Long Short-Term Memory Kalman Filters:Recurrent Neural Estimators for Pose Regularization
Huseyin Coskun, Felix Achilles, Robert DiPietro, Nassir Navab, Federico Tombari
TL;DR
One-shot pose estimates are noisy, while conventional Kalman filters depend on hand-crafted motion and measurement models that can be crude. The paper proposes LSTM-KF, which learns Kalman-filter motion and noise models from data, and reports state-of-the-art performance across three pose-estimation tasks.
Problem
One-shot pose estimates are noisy, while Kalman filters require manually specified motion and measurement models that may be crude approximations of reality.
Method
LSTM-KF learns the Kalman filter’s motion model and noise parameters from data using LSTM modules, while retaining Kalman-style measurement updates.
Results
LSTM-KF achieves state-of-the-art performance across three diverse tasks and outperforms standalone Kalman filters and LSTM.
Takeaways & Limitations
The learned filter regularizes pose estimates without requiring hand-crafted transition or measurement models.
Takeaways & Limitations
Learning-based temporal regularization requires sufficient training data to cover all possible motion paths.
Abstract
from arXiv · showhide
One-shot pose estimation for tasks such as body joint localization, camera pose estimation, and object tracking are generally noisy, and temporal filters have been extensively used for regularization. One of the most widely-used methods is the Kalman filter, which is both extremely simple and general. However, Kalman filters require a motion model and measurement model to be specified a priori, which burdens the modeler and simultaneously demands that we use explicit models that are often only crude approximations of reality. For example, in the pose-estimation tasks mentioned above, it is common to use motion models that assume constant velocity or constant acceleration, and we believe that these simplified representations are severely inhibitive. In this work, we propose to instead learn rich, dynamic representations of the motion and noise models. In particular, we propose learning these models from data using long short term memory, which allows representations that depend on all previous observations and all previous states. We evaluate our method using three of the most popular pose estimation tasks in computer vision, and in all cases we obtain state-of-the-art performance.
1. Introduction
Pose estimation from images is noisy in video, while conventional Kalman filters rely on hand-specified models that may poorly represent real motion. LSTM-KF learns Kalman-filter internals from data and outperforms standalone Kalman filters and LSTM across three pose-estimation tasks.
- Motivation: One-shot pose estimates can be noisy and confuse visually similar but spatially distinct features when temporal information is ignored.The introduction gives left-versus-right legs in body-joint localization as an example.
- Limitations of conventional filtering: Kalman filters are widely used because they are simple and general, but their transition and measurement models are difficult to specify a priori.Common constant-velocity and constant-acceleration assumptions are described as crude approximations to reality.
- Learning-based alternatives: Learned motion models can enrich Kalman filtering, but require enough training data to cover all possible motion paths.This data-coverage constraint is identified as a challenge for learning-based temporal regularization.
- Proposed approach: LSTM-KF learns the Kalman filter’s motion model and noise parameters, enabling temporal regularization without hand-crafted transition or measurement models.The architecture is intended to regularize outputs from generic one-shot estimators.
- Proposed approach: Three LSTMs separately predict the new state, prediction noise, and measurement noise, while Kalman-style updates retain measurement refinement.The decomposition gives each LSTM a defined objective and allows reliance on measurements when predictions are inaccurate.
- Results: LSTM-KF outperforms Kalman filters with different transition models and LSTM on body landmark localization, object tracking, and camera pose estimation.The evaluation uses real data from benchmark datasets.
2. Related Work
Prior temporal-regularization work includes implicit learned schemes, particle filters, static Kalman-parameter optimization, and dynamic parameter regression. LSTM-KF instead combines learned filtering components with a black-box one-shot estimator.
- Implicit temporal regularization: Prior work for human pose estimation used learned dictionaries of joint positions and velocities to regularize estimates through 2D backprojection optimization.The paper reports superior Human3.6M results for its own approach relative to this method.
- Implicit temporal regularization: Object-pose tracking methods used particle filters with predefined constant-velocity motion models, later enhanced with improved features and GPU implementation.These approaches propagate a posterior distribution through time.
- Learning-based Kalman filters: Machine-learning Kalman approaches include static optimization of noise covariances and dynamic regression of transition or noise parameters.The related-work taxonomy distinguishes static parameter learning from active parameter regression during filtering.
- LSTM-KF: LSTM-KF uses three LSTM modules to predict Kalman-filter internals and can be trained end to end through time.Figure 2 depicts both the architecture and its temporal unrolling.
- Learning-based Kalman filters: Some neural approaches jointly learn state propagation and measurement incorporation while assuming constant covariances, rather than using Kalman prediction and update steps.The paper contrasts this design with the standard scheme for linear transitions and additive Gaussian noise.
- LSTM-KF: Unlike methods requiring the one-shot estimator to provide measurement uncertainty, LSTM-KF estimates measurement noise from past observations for black-box estimators.This design enables combination with existing one-shot estimators.
3. Background
Kalman filters provide iterative prediction and measurement updates for sequential estimation, while LSTMs model sequential data using gated memory that can retain information across time.
- Kalman filters are optimal state estimators under linearity and Gaussian-noise assumptions when the model matrices are known.
- The Kalman filter alternates between a prediction step and an update step that incorporates the current measurement through the Kalman gain.
- Recurrent neural networks are suited to sequential data, but early simple RNNs were difficult to train because of vanishing gradients.
- LSTM addresses this training problem with forget, input, and output gates that reset, write, and read recurrent memory.The intermediate memory cell and gates depend on the current input and learned parameters.
- Multiple-layer LSTMs extend the architecture by feeding one layer’s hidden state into the next layer as its input.
4. LSTM Kalman Filters
LSTM-KF replaces fixed Kalman transition and covariance components with learned, dynamic LSTM-based components while retaining standard Kalman prediction and update operations. The architecture combines three LSTM modules and is trained end to end for temporal pose regularization.
- LSTM-KF learns a nonlinear transition function and process and measurement covariances instead of specifying fixed Kalman components.The model assumes incoming measurements are noisy estimates of the underlying state, so H = I.
- Prediction and update steps: The prediction step uses the learned transition function, its Jacobian, and a learned process covariance.
- Prediction and update steps: The update step combines the predicted state, observed measurement, and learned measurement covariance through standard Kalman equations.
- Architecture: Three LSTM modules model the transition, process covariance, and measurement covariance, with inputs derived from prior predictions and observations.
- Architecture: Process and measurement covariance matrices are restricted to diagonal positive-definite forms by exponentiating the corresponding LSTM outputs.
- Training: The model is optimized end to end over parameters from all three LSTM modules using a loss augmented to improve gradient flow to LSTMf.The loss weight λ is set to 0.8 using Human3.6M and fixed for other experiments; training uses backpropagation through time and Adam.
5. Experiments
The experiments evaluate LSTM-KF against standard temporal regularizers across human pose, camera pose, and object tracking. LSTM-KF generally improves estimates, while learned uncertainty adapts to occlusion and the filter combines measurements with learned dynamics.
- Learned filtering behavior: During training, the Kalman gain falls from a high initial value, shifting reliance from measurements alone toward both measurements and LSTMf output.The figure tracks LSTM-KF error and mean Kalman gain during training.
- Experimental setup: LSTM-KF is compared with constant-velocity and constant-acceleration Kalman filters, EMA, and standard LSTM across four RGB-image datasets.The datasets cover 3D human pose, camera pose, and object pose estimation.
- Learned filtering behavior: The model assigns high measurement uncertainty to frames where the left hand is occluded.Figure 5 reports the normalized Euclidean norm of covariance coefficients during a Walking test sequence.
- Human pose estimation: 14% average improvement over the best state-of-the-art approach is achieved on Human3.6M, while LSTM-KF outperforms raw measurements and standard LSTM across actions.Qualitative results particularly show improvements for arm and leg joints.
- Camera pose estimation: LSTM-KF improves Cambridge Landmarks by up to 6.23% for translation and 7.53% for rotation on average, with exceptions for King’s College and S. Facade.The King’s College training and test trajectories differ, while S. Facade has only 231 training frames.
- Object tracking: LSTM-KF reaches 0.72 mm average object-position error, improving 47.05% over the original estimation.Kalman Vel, Kalman Acc, and EMA did not meaningfully improve translation, while standard LSTM deviated from the correct trajectory.
6. Conclusions
The paper introduces LSTM-KF to learn Kalman-filter motion and noise models from data rather than specifying them a priori. Across diverse temporal-regularization tasks, it outperforms standalone Kalman filters and LSTM, achieving state-of-the-art results.
- Contribution: LSTM-KF learns rich motion and noise models from data, relieving the modeler from specifying them a priori.The architecture is designed for temporal regularization of one-shot estimates.
- Results: LSTM-KF outperforms both standalone Kalman filters and standalone LSTM for temporal regularization across three diverse tasks.The paper reports state-of-the-art performance in its experiments.
- Results: 13.8% lower joint error on Human3.6M reduces the error from 82.3 mm to 71.0 mm.This is given as an example of the reported state-of-the-art performance.