End-to-end Learning of Multi-sensor 3D Tracking by Detection

I. INTRODUCTION

• Multi-target tracking remains open amid truncation, fast targets, lighting, sensor motion, occlusion, and path crossing.The task identifies objects in each frame and links their trajectories over time.Source: §I · p. 1
• Tracking by detection typically detects candidate objects first, then associates them over time, with tracking and detection often formulated in 2D.Appearance and motion cues are commonly used for association.Source: §I · p. 1
• LIDAR provides accurate spatial positioning for 3D tracking, while camera images offer richer appearance information than LIDAR.The passage contrasts LIDAR-based tracking with filtering methods common in robotics.Source: §I · p. 1
• The proposed model combines camera and LIDAR data for detection and matching, with end-to-end learning and exact inference via feedforward passes and a linear program.Its matching cost uses appearance and motion representations processed by a siamese network.Source: §I · p. 1

II. RELATED WORK

• Unlike hand-crafted similarity measures, the proposed siamese network learns feature representations and similarity from appearance and 3D spatial cues.Its 3D object detector supplies 3D bounding boxes for matching.Source: §II · pp. 1–2
• The authors contrast their approach with concurrent end-to-end tracking work, stating that structured hinge loss backpropagated through a linear program simplifies the problem and yields better experimental results.Source: §II · p. 2

III. DEEP STRUCTURED TRACKING

• The deep structured model uses feedforward neural networks to compute detection and matching factors from camera and LIDAR data.Learning is end-to-end via structured hinge loss, jointly optimizing the detector and tracker.Source: §III · p. 2

A. Model Formulation

• Given candidate detections across a sequence, the model estimates which are true positives and links them into trajectories.The number of targets can vary as objects appear and disappear from view.Source: A. Model Formulation · p. 2
• The integer linear program uses binary variables for detection status, links between detections, and trajectory starts and ends.Start and end variables allow detections near the LIDAR range limit or under heavy occlusion to mark trajectory boundaries.Source: A. Model Formulation · p. 2
• The objective assigns costs to variable assignments, while constraints rule out assignments that do not form valid trajectories.All detection, link, entry, and exit variables are collected into a single vector.Source: A. Model Formulation · p. 2

B. Conservation of Flow Constraints

• Flow constraints prevent overlapping trajectories and require active detections to connect to a preceding detection or start a trajectory.A detection can end only when active and not linked to a detection in the next frame; candidate matches are drawn from adjacent frames.Source: B. Conservation of Flow Constraints · p. 3

C. Deep Scoring and Matching

• The detection network classifies MV3D's oriented 3D object proposals using camera-image patches projected from their 3D boxes.Proposals reduce the search space by identifying regions likely to contain vehicles.Source: C. Deep Scoring and Matching · p. 3
• The matching network combines an appearance-processing convolutional branch with feedforward branches for 2D and 3D spatial information.Their activations feed a fully connected layer that produces the matching score.Source: C. Deep Scoring and Matching · p. 3
• Appearance features come from a shared-weight siamese network, which uses skip-pooling to retain both low-level and higher-level features.The detections are resized to 224 × 224, and the representation contains five activations.Source: C. Deep Scoring and Matching · p. 3
• Spatial scoring uses bird's-eye 3D occupancy and frontal-view 2D occupancy grids, with IMU-based ego-motion correction aligning successive frames.Fully connected layers preserve location-dependent spatial patterns that a spatially invariant convolutional architecture would lose.Source: C. Deep Scoring and Matching · pp. 3–4

D. Inference

• Total unimodularity allows the constrained integer tracking problem to be relaxed to a linear program while guaranteeing optimal integer solutions.Source: D. Inference · p. 4
• Before solving the linear program, the system computes detection costs for each detection and link costs for each linkable pair.Trajectory start and end costs are learned scalars and require no additional computation.Source: D. Inference · p. 4
• Solving the linear program after computing costs yields the global optimal solution across all frames.Alternative min-cost-flow formulations are stated to give the same solution as the linear program.Source: D. Inference · p. 4

E. End-to-End Learning

• The end-to-end training algorithm backpropagates through a linear program using a structured hinge loss.The authors describe this as a central contribution because it enables joint training of tracking by detection.Source: E. End-to-End Learning · p. 4
• The task loss uses Hamming distance between inferred variable values and ground-truth assignments to reflect that mistakes differ in severity.The model is trained using subgradient descent; loss-augmented inference yields y*, and the decomposed loss makes that inference another exactly solvable linear program.Source: E. End-to-End Learning · p. 4
• Table I compares tracking results from end-to-end and piecewise learning approaches.The supplied table passage states the comparison’s scope but provides no result values.Source: Table I · p. 5
• Table II reports results on the KITTI test set.The supplied table passage provides no result values.Source: Table II · p. 5

IV. EXPERIMENTAL EVALUATION

• The evaluation uses the MV3D train/validation split for consistent validation results because MV3D supplies the detector.The section covers both model performance and training details.Source: §IV · p. 5

A. Dataset

• KITTI contains 40 sequences, split into 20 for training/validation and 20 for testing, with vehicles annotated in 3D.The training set contains 8,026 images and 30,601 vehicles with varying truncation and occlusion.Source: A. Dataset · p. 5
• Unique IDs for annotated 3D trajectories allow the authors to infer trajectory starts, ends, and detection links, establishing ground-truth binary-variable assignments.Source: A. Dataset · p. 5

B. Metrics

• Matching performance is measured by network accuracy when detections are matched between consecutive frames.Source: B. Metrics · p. 5
• MOTA measures trajectory-configuration errors from misses, false positives, and mismatches, while MOTP measures total estimated-position error for object-hypothesis pairs.Source: B. Metrics · p. 5
• The KITTI tracking metrics are computed in 2D and do not evaluate the 3D positioning of trajectories.The evaluation includes MT/ML, CLEAR MOT, ID-Switches, fragmentations, and false positives.Source: B. Metrics · p. 5
• MT marks trajectories covered for more than 80% of ground-truth length, while ML marks coverage below 20%.These metrics report the percentage of trajectories in each category.Source: B. Metrics · p. 5

• Adam uses a 10^-5 learning rate, β1 of 0.9, and β2 of 0.999.Source: C. Training Parameters · p. 5
• CNNs start from ImageNet-pretrained VGG16 weights, while fully connected layers, including weights for binary variables y, use a truncated normal with mean 0 and standard deviation 10^-3.Source: C. Training Parameters · p. 5
• Initialization samples beyond 2 standard deviations from the mean are discarded and redrawn.Source: C. Training Parameters · p. 5

• End-to-end training improves precision and accuracy over piecewise training but decreases coverage, reflecting a trade-off with shorter, more accurate trajectories.The authors say this trade-off is preferable in autonomous driving because merging tracks on one vehicle can produce inaccurate velocity estimates and possible collisions.Source: D. Experiments · p. 6
• In KITTI, the approach is competitive with state of the art, best for MOTP and MT, and second best for ML.Source: D. Experiments · p. 6
• Although the tracker performs 3D tracking, KITTI evaluates metrics in 2D, which does not fully represent the approach's performance.Source: D. Experiments · p. 6
• Figure 5 visualizes tracker trajectories over 15 frames, with the same color marking the same object.Source: Figure 5 · p. 7
• The proposed matching approach significantly outperforms the evaluated image-space and spatial affinity metrics.Affinity thresholds were selected by cross validation using the same train/validation split as the matching network.Source: D. Experiments · pp. 6–7
• Matching becomes harder with occlusion through 50% and then settles; distant vehicles are harder to match, and failures also occur with truncation and poor lighting.The 2D bounding-box size histogram supports the distance finding because box size is directly correlated with 3D distance.Source: D. Experiments · p. 7
• Relative matching error stays close to constant across detection-overlap levels, suggesting robustness to bounding-box tightness.Source: D. Experiments · p. 7

V. CONCLUSIONS

• The deep structured model uses feedforward neural nets with camera and LIDAR data to compute detection and matching scores.Source: §V · p. 7
• On KITTI, the approach is very competitive and outperforms the state of the art in the MOTP and MT metrics.Source: §V · p. 8
• Inference is exact, using feedforward processes followed by solving a linear program.Source: §V · p. 8
• Future work is to extend the approach to handle long-term occlusions and missing detections.Source: §V · p. 8