MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries

• Multi-camera tracking must contend with uncertain 3D detection, occlusion, appearance ambiguity, and objects crossing camera boundaries.These challenges call for spatially consistent predictions across cameras as well as temporal association.Source: §1 · p. 1
• MUTR3D introduces 3D track queries that model track states and appearance across time and cameras, sampling features from visible cameras and learning to create, track, or end tracks.Objects decoded from the same queries across frames are inherently associated.Source: §1 · p. 2
• The framework is presented as the first fully end-to-end multi-camera 3D tracking method, implicitly modeling track position and appearance variances without common post-processing.The listed steps eliminated include non-maximum suppression, bounding-box association, and object re-identification.Source: §1 · p. 2
• 27.0% AMOTA on the nuScenes vision-only 3D tracking dataset, with 12% less ID switch than previous SOTA methods in the multi-camera setting.Source: §1 · p. 2
• Average Tracking Velocity Error (ATVE) and Tracking Velocity Error (TVE) evaluate motion-model error in the current 3D tracker.Source: §1 · p. 2

2.1. 3D MOT in Autonomous driving

• Camera-based 3D MOT combines appearance, motion, and depth-ordering cues, but still trails LiDAR-based tracking on the nuScenes public leaderboard: 21.7% versus 67.9% AMOTA.Prior methods include Kalman-filtered location prediction with 3D IoU, learned motion or appearance cues, graph neural networks, and life-cycle management.Source: §2.1 · pp. 2–3

2.2. Camera-based 3D Detection

• Camera-based 3D detection methods range from 2D-plane approaches to 3D-space or bird’s-eye-view representations, including inverse-projection query-based detection.The text notes that performance advantages of working in 3D space remain unclear and need further comparison.Source: §2.2 · p. 3

2.3. Query based detection and tracking

• MUTR3D builds on query-based detection and tracking by extending the framework to multi-camera 3D MOT with a motion model.The related work traces query-based set prediction from DETR to online 2D trackers.Source: §2.3 · p. 3

3.1. Query based Object tracking

• Track queries represent whole tracklets, update frame by frame, and directly associate same-query detections across frames for online joint detection and tracking.The tracker combines a query-based loss, multi-camera sparse attention, and a motion model.Source: §3.1 · p. 3

3.2. End-to-end object tracking loss

• Newborn queries are matched to newly appeared objects, while old queries retain their first successful ground-truth assignment or map to no object when absent.The loss sums paired box losses over the one-to-one assignment between candidates and padded ground truths.Source: §3.2 · pp. 3–4

3.3. Multi-camera Track query decoding

• 3D reference points project each query into visible cameras for feature sampling, and the extracted features update queries and their reference points before candidate decoding.The decoder uses self-attention between queries and cross-attention with multi-camera image features.Source: §3.3 · pp. 4–5

3.4. Query Life Management

• Query life management removes inactive queries using predicted-box classification scores, with separate thresholds for newborn and old queries and a time-length parameter.Old queries are removed when their scores remain below the old-query threshold for the specified successive frames.Source: §3.4 · p. 5

3.5. Query Update and Motion model

• After filtering dead queries, MUTR3D updates reference points to model object dynamics and compensate for ego-motion, using query-predicted velocity across frames.A small FFN predicts ego-frame velocity and is supervised with ground truth; a memory bank also supplies previous-frame features for query updates.Source: §3.5 · p. 5

4.1. Datasets

• The nuScenes experiments use 700 training, 150 validation, and 150 test sequences, with six cameras and roughly 40 annotated keyframes per sequence.The keyframes are synchronized across sensors at 2 FPS and include annotations for seven object categories.Source: §4.1 · p. 5

4.2. Evaluation Metrics

• AMOTA and AMOTP integrate tracking accuracy and precision over recall levels, with AMOTA’s per-recall MOTA accounting for false positives, false negatives, and identity switches.AMOTP uses matched-track 2D bird’s-eye-view position error; additional reported metrics include MOTA, MOTP, and IDS.Source: §4.2 · pp. 5–6

4.4. Compare with State-of-the-art

• MUTR3D exceeds QD3DT by 0.052 AMOTA on validation and 0.053 on test, while operating without NMS or an association stage.Against tailored SimpleTrack Kalman baselines, it achieves only slight gains on AMOTA, MOTA, and MOTP.Source: Table 1 · p. 6, §4.4 · p. 6

4.3. Implementation Details

• The tracker uses a ResNet-101 feature extractor and DETR3D-pretrained weights, and trains for 72 epochs on three-frame clips.The paper also evaluates Kalman-filter baselines using the DETR3D detector, including a SimpleTrack-based variant.Source: §4.3 · p. 6

4.5. Evaluating Motion Models

• MUTR3D estimates velocity more accurately than QD3DT and achieves better TVE than Kalman-filter motion models.ATVE and TVE evaluate tracking velocity error; ATVE averages matched predicted-versus-ground-truth velocity L2 error over recall levels.Source: §4.5 · p. 7, Table 3 · p. 7

4.6. Ablation study

• Removing the motion model, which updates 3D reference points, degrades performance across all reported metrics.Training with 3, 4, or 5 frames shows gradually improving performance as the number of frames increases.Source: §4.6 · p. 7, Table 4 · p. 7, Table 5 · p. 7

4.7. Qualitative results

• In an eight-second qualitative clip, MUTR3D correctly tracks near-field cars truncated across multiple camera views.The examples include gray and black cars truncated in the Front-Left and Back-Left cameras.Source: §4.7 · p. 7

5. Conclusion

• MUTR3D outperforms QD3DT [11] by 5.3 AMOTA and 4.7 MOTA on the nuScenes test dataset.Source: §5 · p. 7
• The end-to-end framework performs 3D detection, compensates for ego-motion and object motion, and associates objects across cameras and frames.Source: §5 · p. 7
• The paper introduces Average Tracking Velocity Error (ATVE) and Tracking Velocity Error (TVE) to evaluate motion models in 3D trackers.Source: §5 · p. 7
• Figure 3 shows eight consecutive frames across four views, using consistent colors for identities and arrows for estimated velocity.The example includes objects truncated across cameras.Source: Figure 3 · p. 8