Source-linked AI summary
Deep Tracking: Seeing Beyond Seeing Using Recurrent Neural Networks
Peter Ondruska, Ingmar Posner
TL;DR
The paper addresses object tracking from partial, occluded sensor observations without hand-engineered models or ground-truth scene states. It uses recurrent neural networks and unsupervised training to predict complete scene states, demonstrating faithful reconstruction on a synthetic 2D sensing task.
Problem
Object tracking in partially observable dynamic scenes commonly requires hand-engineered pipelines and ground-truth states containing unseen regions.
Method
Deep Tracking uses recurrent neural networks to map raw sensor sequences to unoccluded scene estimates while learning hidden-state dynamics end to end.
Results
The system predicts object positions in real time through complete occlusions and noise, achieving highly faithful reconstructions on a synthetic 2D sensing dataset.
Takeaways & Limitations
The approach provides an end-to-end tracking solution that learns plant and sensor models directly from raw, occluded sensor data.
Takeaways & Limitations
The demonstrated approach remains limited to a simulated 2D sensing application; evaluation on real robot data and other modalities is future work.
Abstract
from arXiv · showhide
This paper presents to the best of our knowledge the first end-to-end object tracking approach which directly maps from raw sensor input to object tracks in sensor space without requiring any feature engineering or system identification in the form of plant or sensor models. Specifically, our system accepts a stream of raw sensor data at one end and, in real-time, produces an estimate of the entire environment state at the output including even occluded objects. We achieve this by framing the problem as a deep learning task and exploit sequence models in the form of recurrent neural networks to learn a mapping from sensor measurements to object tracks. In particular, we propose a learning method based on a form of input dropout which allows learning in an unsupervised manner, only based on raw, occluded sensor data without access to ground-truth annotations. We demonstrate our approach using a synthetic dataset designed to mimic the task of tracking objects in 2D laser data -- as commonly encountered in robotics applications -- and show that it learns to track many dynamic objects despite occlusions and the presence of sensor noise.
Introduction
Deep Tracking addresses tracking in partially observable, dynamic scenes by replacing hand-engineered pipelines with an end-to-end recurrent model that maps raw sensor sequences to complete scene estimates.
- Introduction: Robots often observe only parts of complex, dynamic environments, although decision making typically requires complete situational awareness.
- Introduction: Conventional tracking pipelines separate detection and tracking and require hand-engineered plant, observation, and data-association models.
- Introduction: Deep Tracking maps raw sensor data directly to an interpretable representation of a complete, unoccluded scene.
- Introduction: RNNs update a learned hidden belief state from partial observations and decode it into an interpretable unoccluded scene without hand-engineering.
- Introduction: The system is trained unsupervised from raw sensor data and predicts all object positions in real time through complete occlusions and noise.
Deep Tracking
The model predicts an unoccluded scene from sequential partial observations by representing world dynamics with a hidden Markov state and recursively updating its belief.
- Deep Tracking: The objective is to model P(y_t|x_1:t), predicting the current unoccluded scene from the sequence of sensor observations.
- Deep Tracking: Observed scene elements are encoded with measurements, while unobserved elements receive {0, 0}; changing viewpoints expose different scene parts over time.
- Deep Tracking: Temporal and structural regularities, including object motion and appearance, support prediction during later occlusions.
- Deep Tracking: Deep tracking uses RNNs to model P(y_t|x_1:t) from sequential sensor observations.
- The Model: The scene process y_t is not Markov because positions omit information such as speed and acceleration needed for prediction.
- The Model: An underlying Markov process h_t is assumed to capture the complete world state, with P(y_t|h_t) modeling unoccluded sensor space and P(x_t|y_t) the sensing process.
- The Model: Recursive Bayesian estimation maintains Bel(h_t)=P(h_t|x_1:t), first predicting a belief and then correcting it with the latest measurement.
neural network
The approach uses a recurrent neural network to update a learned belief state from partial observations and decode the current or future unoccluded scene. It avoids explicitly specifying belief representations and generative-process distributions by learning both mappings end-to-end.
- neural network: The filtering process uses recurrent memory to map sequences of partial sensor observations into an estimate of the complete scene.The paper illustrates this process as a recurrent mapping from observations and memory to belief state and scene prediction.
- neural network: The recurrent mapping updates belief state B_t from the previous belief B_t−1 and sensor observation x_t.The belief state is passed between time steps as the network’s memory.
- neural network: The network models P(y_t|B_t) to decode the estimated unoccluded scene from the learned belief state.Future states can also be predicted by supplying empty observations for subsequent time steps.
- neural network: The method replaces explicit belief-state and generative-process specifications with differentiable neural networks learned directly from data.The networks for belief updates and scene decoding are trained together end-to-end.
Training
The paper introduces unsupervised training using only raw, occluded sensor observations rather than ground-truth scene states. Future-state prediction with spatial and temporal input dropout forces the network to learn object dynamics during occlusion.
- Training: Supervised training requires ground-truth scene states containing portions that the robot sensor cannot observe, making data collection impractical or costly.The proposed alternative uses only a sufficiently long record of raw sensor observations.
- Training: Unsupervised training learns the belief-update and decoding functions using only the observed sensor sequence x_1:t.Observed sensor measurements provide labels for only directly visible parts of the scene.
- Training: The network predicts a future state P(y_t+n|x_1:t) instead of the current state, requiring it to model object dynamics through unseen intervals.The learned dynamics are then used to predict the future scene after occlusion.
- Training: Training drops observations across space and multiple time steps while retaining visible future targets, preventing the network from simply copying inputs.This input-dropout scheme is intended to force learning of the correct temporal and spatial patterns.
Experimental Results
The experiment evaluates a four-layer recurrent network on simulated 2D laser data with multiple dynamic objects. It learns scene completion and tracking through occlusion, including under altered object shapes and limited sensor noise.
- Experimental Results: The simulated environment contains 2–12 independently moving objects around a robot equipped with a 2D laser scanner.Objects are circles moving at constant velocity in random directions and appear or disappear over time.
- Experimental Results: The sensor input is a stream of 50×50 2-channel binary images encoding pixel visibility and visible obstacles.The scene is represented on a 50×50 grid, and observations are generated by ray-tracing visibility.
- Experimental Results: The experiment uses a 4-layer recurrent architecture in which the Encoder detects visible objects, the Belief tracker updates B_t, and the Decoder performs scene deocclusion.The network has 11k parameters and maintains B_t in a 50×50×16 third-layer representation.
- Experimental Results: One network pass takes 10ms on a standard laptop, supporting real-time data filtering.The final layer represents a probabilistic occupancy grid, with training based on binary cross-entropy.
- Experimental Results: Unsupervised and supervised training produced almost identical results, while the trained network tracked objects through long periods of complete occlusion after seeing only part of an object.Training first learned visible object shapes and later learned position tracking through occlusions.
- Experimental Results: The learned belief state represented hypotheses with different motion patterns through distinct activation patterns tracked across frames.These representations were not hard-coded but emerged from adaptation to the task.
- Experimental Results: The network correctly learned and predicted world state when object shapes changed from circles to squares and when sensor noise affected 1% of pixel observations.The authors note that more complex cases may require deeper networks or different recurrent units such as LSTM.
Related Works
The paper positions Deep Tracking against hand-designed Bayesian and neural tracking pipelines by learning belief representations and update operations end to end for partially observable dynamic scenes.
- Classical tracking: Bayesian filtering commonly uses hand-designed state representations and tractable prediction and correction operations based on distributional assumptions or sampling.The Kalman filter is cited as an example that represents belief with a multivariate normal distribution.
- Deep tracking: Deep Tracking instead learns an appropriate belief state and corresponding prediction and update operations for multiple objects with varied appearance and complex behavior.This contrasts with hand-designed pipelines and supports end-to-end learning.
- Learned representations: Figure 6 shows encoder spikes at visible-object positions and distinct belief-state activation patterns for objects with different motion patterns.The figure reports activations from different network parts, with one filter per layer.
- Neural tracking challenges: Full neural tracking must simulate latent system dynamics when measurements are absent, such as during object occlusion.The challenge involves modeling high-dimensional distributions for complex dynamic scenes.
- Neural tracking challenges: Temporal Restricted Boltzmann Machines model high-dimensional sequence distributions but require sampling, making inference and learning computationally expensive.The paper instead directly models P(y|x) with feed-forward recurrent networks, enabling exact inference and gradient computation through standard back-propagation.
Conclusions
Deep Tracking maps raw sensor data to an interpretable hidden sensor-space representation and predicts the unoccluded state of an entire simulated 2D scene. It achieved highly faithful reconstructions on synthetic data, while evaluation on real robotic data remains future work.
- Conclusion: Deep Tracking uses recurrent neural networks to map raw sensor data to an interpretable hidden sensor-space representation and predict the entire scene's unoccluded state.The demonstrated application is simulated 2D sensing.
- Conclusion: The method avoids hand-crafted plant and sensor models by learning them directly from raw, occluded sensor data.This is presented as an end-to-end approach to scene reconstruction.
- Conclusion: Highly faithful reconstructions of the underlying world model were achieved on a synthetic dataset.The conclusion reports this result for the simulated 2D sensing application.
- Future work: Future work will evaluate Deep Tracking on real robot data in pedestrianized areas and autonomous driving with other traffic participants.The authors also intend to extend the approach to 3D point clouds and depth cameras.