Source-linked AI summary
Online Multi-Target Tracking Using Recurrent Neural Networks
Anton Milan, Seyed Hamid Rezatofighi, Anthony Dick, Ian Reid, Konrad Schindler
TL;DR
Online multi-target tracking must jointly estimate continuous target states, associate measurements, and manage an unknown, changing number of targets. The paper proposes an end-to-end recurrent architecture that performs these tasks with learned components, obtaining encouraging results on simulated and real data at approximately 300 Hz on a standard CPU.
Problem
Online multi-target tracking remains difficult because it combines continuous state estimation, combinatorial data association, and an unknown, time-varying number of targets.
Method
A unified recurrent neural network performs prediction, data association, state updates, and target initiation and termination, using modular blocks and learned target-existence and assignment representations.
Results
Qualitative and quantitative experiments on simulated and real data show encouraging results, with execution at approximately 300 Hz on a standard CPU.
Takeaways & Limitations
The results support the potential of end-to-end recurrent learning for online multi-target tracking in realistic scenarios.
Takeaways & Limitations
The presented formulation uses network order N = 1, assuming targets move independently and reusing the same RNN for each target.
Abstract
from arXiv · showhide
We present a novel approach to online multi-target tracking based on recurrent neural networks (RNNs). Tracking multiple objects in real-world scenes involves many challenges, including a) an a-priori unknown and time-varying number of targets, b) a continuous state estimation of all present targets, and c) a discrete combinatorial problem of data association. Most previous methods involve complex models that require tedious tuning of parameters. Here, we propose for the first time, an end-to-end learning approach for online multi-target tracking. Existing deep learning methods are not designed for the above challenges and cannot be trivially applied to the task. Our solution addresses all of the above points in a principled way. Experiments on both synthetic and real data show promising results obtained at ~300 Hz on a standard CPU, and pave the way towards future research in this direction.
Introduction
The paper develops an end-to-end RNN approach for online multi-target tracking, addressing variable target counts, continuous state estimation, and data association. It combines recurrent components for tracking tasks and reports encouraging results on simulated and real data.
- Motivation: Multi-target tracking must handle changing numbers of targets, continuous state estimation, and discrete data association in varied real-world scenes.The scene variation includes viewpoint, camera motion, lighting, occlusion, and target density.
- Motivation: Deep learning has been little used for multi-target tracking because training data is limited and the task combines variable-length sequences with discrete and continuous variables.Earlier recurrent approaches were demonstrated under restricted simulated or occupancy-estimation settings without explicit data association.
- Contributions: The proposed unified RNN performs prediction, data association, state update, and target initiation and termination within one model-free network structure.The approach does not require prior knowledge of target dynamics or clutter distributions and can represent linear, nonlinear, and higher-order dependencies.
- Contributions: The learned data-association component handles target birth and death while inferring unordered sets with unknown cardinality.This extends recurrent models beyond fixed-sized input and output vectors.
- Contributions: A generative model provides arbitrary amounts of training data through sampling.
- Results: Qualitative and quantitative experiments on simulated and real data show encouraging results and support the approach’s potential.
Related Work
Prior multi-object tracking uses sophisticated probabilistic and appearance-based models, while deep learning has expanded in other domains but remains constrained by data and output-format requirements. RNNs offer sequence modeling capabilities, yet multi-target tracking introduces additional state and variable-structure challenges.
- Multi-object tracking: Earlier multi-object tracking methods include multiple hypothesis tracking and joint probabilistic data association, originally developed for radar and sonar.
- Deep learning: Deep learning methods historically required very large training datasets to avoid overfitting their many parameters.
- Deep learning: CNNs achieve strong results on many applications but impose restrictive output formats.
- Recurrent neural networks: RNNs map input sequences to arbitrary output sequences and provide memory through recurrent connections, given known alignment and dimensions.
- Recurrent neural networks: Multi-target tracking is harder for RNNs because each time step contains multidimensional states combining continuous and discrete variables.
Background
RNNs sequentially predict outputs from prior hidden states, while LSTMs add gated memory for more difficult data association. Bayesian filtering provides the classical prediction-and-update framework, with multi-target tracking adding association and track-management problems.
- Recurrent neural networks: An RNN makes a prediction at each time step from the previous state and possibly an additional input.
- Recurrent neural networks: The hidden state is the main control mechanism, with the final layer representation used to produce the desired output.
- Long short-term memory: LSTMs augment the hidden state with a memory representation controlled by gates that forget or replace information.
- Bayesian filtering: Bayesian filtering recursively estimates the true state from noisy measurements under a Markov assumption.
- Bayesian filtering: Filtering typically alternates between predicting state dynamics and updating beliefs using current measurements.
- Multi-target challenges: Multiple-target tracking additionally requires measurement-to-target association and mechanisms for spawning and removing tracks as targets appear or disappear.
Our Approach
The approach casts Bayesian state estimation, data association, and track management as recurrent networks trained end to end. It represents target states and measurements explicitly, models target existence and assignments, and separates tracking from association into modular blocks.
- Core formulation: The method converts state estimation, data association, and track initiation and termination into a recurrent neural network for end-to-end learning.
- State representation: Each target state is represented by bounding-box coordinates (x, y, w, h), with the formulation extendable to dimensions such as velocity, acceleration, or appearance.
- State representation: The network’s order N captures spatial dependencies between targets; setting N = 1 assumes independent target motion and reuses the same RNN for each target.
- Data association: The assignment matrix gives each target a probability distribution over measurements, including an extra column for missing measurements.
- Track management: The existence vector represents target existence probabilities and supports an unknown, time-varying number of targets.
- Architecture: The framework separates state prediction and update with track management from data association, enabling modular replacement and separate pretraining of blocks.Separate pretraining significantly speeds learning and is necessary in practice for convergence.
Target Motion
The paper uses a temporal RNN to learn target dynamics, predict and update states, and estimate track existence within a unified online framework. Its loss combines trajectory errors with birth/death regularization while avoiding reliance on hand-specified motion or clutter models.
- Target Motion: The temporal RNN learns target dynamics and outputs predicted states, updated states, existence probabilities, and temporal existence changes.These outputs support prediction, state correction, and birth/death decisions across frames.
- Target Motion: Prediction learns target motion without measurements, while update corrects the state distribution using target-to-measurement assignments.The architecture separates these two functions within the recurrent tracking block.
- Target Motion: The RNN-based architecture jointly performs state prediction, state update, and target existence probability estimation.The corresponding architecture is shown on the left side of Figure 2.
- Target Motion: The loss combines trajectory MSE terms with birth/death and regularization terms, and averages each training sample over all sequence frames.The formulation targets predictions and updates that remain close to ground-truth trajectories.
Initiation and Termination
The method represents the changing target count with existence probabilities, enabling track initiation and termination. A smoothness prior reduces overly hard frame-by-frame decisions when measurements are missing.
- Initiation and Termination: Existence probabilities E represent whether each target is present, allowing the tracker to handle an unknown and time-varying number of targets.At test time, targets with E below 0.6 are discarded.
- Initiation and Termination: Binary cross entropy trains existence predictions because they are necessary for target initiation and termination.The BCE term approximates the existence probability for each target.
- Initiation and Termination: A pairwise smoothness prior minimizes absolute differences between consecutive existence probabilities.It is added because BCE alone can produce hard decisions and terminate tracks whenever a measurement is missing.
Data Association with LSTMs
The paper uses an LSTM to learn data association from training data, addressing the discrete one-to-one assignment problem. Synthetic tracking results also expose delayed track initiation and termination as an online limitation.
- Data Association with LSTMs: Data association is a difficult combinatorial task because greedy methods can fail in clutter and joint hypothesis methods are NP-hard.Practical systems therefore commonly rely on efficient approximations.
- Data Association with LSTMs: The LSTM architecture learns data association from training data while enforcing the one-to-one assignment constraint.The authors demonstrate this capability by replicating the linear assignment problem.
- Data Association with LSTMs: Synthetic sequences with clutter evaluate five targets with random birth and death times, including reconstructed trajectories and existence probabilities.The displayed example is a 20-frame sequence.
- Data Association with LSTMs: Delayed initiation and termination appear in the online synthetic result, which the paper identifies as unavoidable for purely online tracking.The delay is visible for the top-most track in the reconstructed trajectories.
- Data Association with LSTMs: The LSTM predicts assignments one target at a time while using the full pairwise-distance matrix between predicted targets and measurements.Its memory and nonlinear transformations are used to learn assignment structure from data.
- Data Association with LSTMs: Figure 5 measures MOTA while varying two exemplar hyper-parameters, but evaluates only the prediction/update block for one target.The single-target setup explains the relatively low MOTA shown in the figure.
Training Data
Because publicly available labelled pedestrian-tracking data is limited by costly video annotation, the paper uses synthetic generation for training data.
- Training Data: The paper resorts to synthetic trajectory generation because only limited labelled pedestrian-tracking data is publicly available.The stated obstacle is the tedious and time-consuming annotation of video.
Implementation Details
The implementation uses separate recurrent modules for state estimation, track management, and data association, trained with RMSprop on approximately 100K short sequences.
- Implementation: The implementation is provided in Lua and Torch7, with the complete code base and pre-trained models publicly available.
- Network size: The state-estimation and track-management RNN has one layer with 300 hidden units, while the data-association LSTM has two layers with 500 hidden units.
- Optimisation: RMSprop training starts at a learning rate of 0.0003, decreases it by 5% every 20,000 iterations, and runs for at most 200,000 iterations.Training both modules takes approximately 30 hours on a CPU.
- Data: The RNN is trained on approximately 100K sequences, each 20 frames long, using mini-batches of 10 samples.Inputs are normalised to [−0.5, 0.5] with respect to image dimensions.
Experiments
Experiments evaluate the approach on synthetic sequences and the MOTChallenge 2015 pedestrian benchmark, including baseline and online-method comparisons. The method is strictly online, uses detector geometry without visual features, and is substantially faster than the highest-accuracy comparison.
- Synthetic data: Synthetic experiments generate five targets with random birth and death times in a cluttered environment, displaying initiation and termination indicators.
- Real-world benchmark: MOTChallenge 2015 contains 22 video sequences split evenly into training and testing, with variation in target motion, camera motion, viewing angle, and person density.Evaluation reports precision, recall, trajectory recovery, false positives, false negatives, identity swaps, fragmentations, MOTA, and MOTP.
- Baseline comparison: The baseline study compares Kalman-Hungarian variants, JPDAm, and two learned variants combining RNN/LSTM components for motion modeling and data association.The learned model performs favourably against the purely online Kalman-HA solution and keeps up with similar approaches.
- Benchmark comparison: The benchmark comparison uses provided detections and contrasts the strictly online method with methods that use offline processing or future frames.The proposed method computes and fixes the solution for each incoming frame before processing the next.
- Feature setting: The method does not use target appearance or optical flow, relying solely on geometric locations supplied by the detector.The authors note that such visual features are often unavailable in applications such as cell or animal tracking.
- Results: Two orders of magnitude faster than the top-accuracy comparison, the approach does not quite reach the highest pedestrian online-tracking accuracy.Figure 6 shows example test-set frames from ADL-Rundle-3, with bounding-box colours indicating person identity.
Discussion and Future Work
The paper frames recurrent networks as a unified solution for online multi-target tracking and highlights learned one-to-one assignment as a non-trivial capability. It identifies appearance and more robust association as directions for improving results.
- Discussion and Future Work: The approach addresses data association and trajectory estimation within a neural-network setting for online multi-target tracking.
- Discussion and Future Work: An RNN-based approach can learn complex motion models in realistic environments, while an LSTM can learn one-to-one assignment.The paper describes one-to-one assignment as non-trivial for this architecture.
- Discussion and Future Work: Incorporating appearance and learning a more robust association strategy are proposed as ways the results could be improved significantly.