Deep Learning on Monocular Object Pose Detection and Tracking: A Comprehensive Overview

1 INTRODUCTION

• Pose estimates have applications in autonomous driving, robotics, and augmented reality, including collision avoidance, robot interaction, and real-to-image projection.The paper describes these applications as uses of positional and orientational information from estimated poses.Source: §1 · p. 2
• The survey covers instance-level and category-level pose detection and tracking, grouping the two tracking subtasks because they have unified outputs and few related works.The authors describe tracking as being at a relatively immature stage; unless otherwise specified, objects refer to rigid objects.Source: §1 · p. 2
• Pose detection and tracking are related: detection can initialize tracking, while tracking is often used to improve detection robustness and smoothness.Category-level tasks are described as more general extensions of instance-level tasks.Source: §1 · p. 2
• The survey distinguishes itself from earlier reviews by focusing on detailed deep learning methods and monocular inputs, and by defining tasks and metrics and comparing datasets.It also reviews pose tracking and analyzes challenges and possible future work, areas few previous surveys covered.Source: §1 · pp. 2–3
• Its scope is a detailed review of deep learning-based object pose detection and tracking methods using monocular RGB/RGBD input.The survey presents qualitative and quantitative state-of-the-art comparisons and analyzes potential future research directions and challenges.Source: §1 · p. 3

2.1 Task Definition

• Instance-level detection recovers a target instance’s camera-relative rotation and translation from a monocular RGB/RGBD image and its CAD model.The pose comprises 6 degrees of freedom, and a trained model detects only its specific object instance.Source: §2.1 · p. 4
• Category-level detection estimates pose and object size from a monocular RGB/RGBD image without a CAD model, and generalizes across instances of a category.Rotation, translation, and size form 9 degrees of freedom; some autonomous-driving works predict 7 by excluding pitch and roll.Source: §2.1 · p. 4
• Monocular tracking recovers poses across a sequence of RGB/RGBD images given an initial pose; object size can be inferred from that initial pose.Tracking can be instance-level, where CAD models may be used, or category-level, where they may not; some autonomous-driving works omit pitch and roll.Source: §2.1 · p. 5

2.2 Metrics Definition

• The 2D projection metric accepts a pose when projected CAD-model points differ by less than 5 pixels, while n°n cm requires both rotation and translation errors below their thresholds.The projection metric is restricted to instance-level methods; n°n cm supports instance- and category-level methods, including symmetry-aware error calculation.Source: §2.2 · p. 5
• ADD(-S) evaluates instance-level methods using model-point distances, with correctness defined below 10% of model diameter; NDS combines mAP and five mean True Positive metrics.For symmetric objects, ADD is often replaced by ADD-S; NDS reflects detection performance and box location, size, orientation, attributes, and velocity.Source: §2.2 · p. 5
• 3DIoU measures overlap between predicted and ground-truth 3D boxes on a normalized 0-to-1 scale, with mAP@IoU25, mAP@IoU50, and mAP@IoU75 commonly reported.These indicators treat poses as correct when 3DIoU exceeds 0.25, 0.5, or 0.75, respectively.Source: §2.2 · pp. 5–6

2 TASKS AND METRICS DEFINITION

• Table 2 presents a comparison of datasets for monocular object pose detection and tracking.Source: Table 2 · p. 6

3 DATASETS FOR MONOCULAR OBJECT POSE DETECTION AND TRACKING

• The dataset review groups benchmarks by instance- or category-level tasks and CAD-model availability, with comparisons summarized in Table 2.Figure 3 provides visual examples.Source: §3 · p. 7

3.1 Datasets for Instance-Level Object Pose Detection and Tracking

• Instance-level coverage spans Linemod for detection, Linemod Occlusion for occlusion testing, and YCB-Video for both detection and tracking.Linemod sequences are short and noncontinuous, so the survey says they are generally used for detection rather than tracking.Source: §3.1 · p. 7
• T-LESS challenges pose estimation with textureless, visually similar electrical parts, while HomebrewedDB includes varied objects, scenes, sensors, occlusion, and appearance changes.HomebrewedDB spans scenes from simple setups to cluttered scenes with heavy occlusion.Source: §3.1 · pp. 7–8

3.2 Dataset for Category-Level Object Pose Detection and Tracking

• Category-level datasets differ in pose coverage: KITTI3D provides 7DoF labels, Apolloscape provides 9DoF annotations, and nuScenes annotates 3D boxes and object attributes.KITTI3D lacks pitch and roll annotations; Apolloscape is described as more suitable than KITTI3D for category-level pose evaluation.Source: §3.2 · p. 8
• NOCS combines synthetic and real RGBD data but has less accurate registered training poses; Objectron's video clips suit AR evaluation, though single dominant objects limit robot-grasping training.Objectron contains mostly one object per image, occupying most of the image area.Source: §3.2 · pp. 8–9

4.1 RGB-based Methods

• RGB-based methods comprise direct, keypoint-based, dense coordinate-based, refinement-based, and self-supervised approaches.Direct regression or classification is lightweight and easy to train, but generally performs worse than correspondence-based methods and can generalize poorly to natural scenes; EfficientPose is described as avoiding refinement.Source: §4.1 · p. 10
• RGB keypoint methods detect image keypoints and solve PnP; voting-based vector fields help recover occluded or truncated keypoints, but methods require predefined keypoints and depend on their detection.Heatmaps can help with occlusion but struggle with truncated objects; richer representations such as HybridPose increase computational cost.Source: §4.1 · pp. 10–11
• RGB dense coordinate methods build dense 2D–3D correspondences for PnP and can handle occlusion and symmetry, but continuous coordinate prediction is harder than sparse keypoint prediction.UV maps simplify correspondence prediction, while EPOS uses compact surface fragments for efficiency and to address global or partial symmetries.Source: §4.1 · pp. 11–12
• RGB refinement aligns rendered and observed images iteratively or with learned update networks, improving pose estimates but adding renderer- and iteration-dependent runtime costs.The survey also describes iNeRF, which inverts an optimized neural radiance field for rendering.Source: §4.1 · p. 12
• RGB self-supervision seeks to reduce annotation costs through synthetic-to-real adaptation or consistency training, but performance remains below fully supervised methods and the domain gap persists.Examples include robot-generated pose labels, unlabeled-real-data fine-tuning, and image or keypoint consistency objectives.Source: §4.1 · pp. 12–14

4.1 RGB-based Methods; 4.2 (RGB)D-based Methods

• RGB methods are compared on Linemod, Occlusion Linemod, and YCB-Video, while (RGB)D comparisons cover keypoint-based and other methods using ADD(-S), not retrieval-based methods.The survey provides no unified quantitative comparison for retrieval-based methods because they do not use common datasets or experimental settings.Source: Table 3 · p. 13, Table 4 · p. 13, Table 5 · p. 13

4.2 (RGB)D-based Methods

• (RGB)D retrieval methods match observed images or depth masks against pose templates or learned descriptors, but discretizing rotation trades prediction precision against computational cost.MTTM uses depth masks and removes irrelevant points, while its performance decreases substantially without CAD models.Source: §4.2 · pp. 14–15
• (RGB)D methods also use 3D keypoint correspondences or specialized architectures: keypoint methods are noise-robust but sensitive to keypoint definitions, while DenseFusion and MoreFusion address clutter and occlusion.Other designs target real-time operation, CAD-free localization, or synthetic-to-real generalization; keypoint approaches may require costly iterative optimization.Source: §4.2 · pp. 15–16
• The schematic presents an overall representation of category-level monocular detection methods.Source: Figure 5 · p. 17
• A performance table covers category-level monocular 3D object detection methods on KITTI3D.Source: Table 7 · p. 17
• A separate performance table covers category-level monocular 3D object detection methods on Nuscenes.Source: Table 8 · p. 17

5.1 Category-Level Monocular 3D Object Detection

• Category-level 3D detection predicts yaw, translation, and size without CAD models, prioritizing translation accuracy for autonomous-driving scenarios.The survey focuses on monocular RGB input rather than LiDAR point clouds.Source: §5.1 · p. 18
• 2D proposal methods lift image detections into 3D, while keypoint methods estimate or refine box keypoints; both face accuracy or efficiency trade-offs.Deep3Dbox [106] uses post-optimization, whereas GS3D [80] enables end-to-end training; keypoint approaches include SMOKE [94] and RTM3D [82], but keypoint errors, occlusion, and nondifferentiable refinement remain concerns.Source: §5.1 · pp. 18–19
• Other 3D approaches include end-to-end parameter fusion, video kinematics, and reinforcement-learning box refinement, with refinement adding computational cost.MonoGRNet [121] fuses 2D detection, center-depth, location, and corner regression; MonoDIS disentangles 3D parameters, while kinematic3D [9] uses monocular video and RAR-Net [90] iteratively changes one box parameter at a time.Source: §5.1 · pp. 18, 19, 21
• Pseudo-LiDAR methods estimate pixel depth and convert it to point clouds, improving accuracy but incurring extra computation and remaining vulnerable to depth noise.Mono3D-PLiDAR [161] addresses pseudo-LiDAR noise, while Peng et al. [117] use real LiDAR during training and DD3D [110] reports state-of-the-art performance after depth pretraining and fine-tuning.Source: §5.1 · pp. 19–21

5.2 Category-Level Monocular 6D Pose Detection

• For 9DoF pose detection, NOCS [153] introduced a shared canonical representation and benchmark dataset, but alignment-based methods rely on reconstruction and pose refinement.SPD [143], ACR-Pose [35], and SGPA address intraclass variation or representation quality; RGB-only detection [76] is possible but performs worse than the RGBD counterparts.Source: §5.2 · pp. 22–23
• Regression methods directly predict pose and size, but their performance is limited by weak prior information, despite architectures designed to represent shape and rotation.CASS [16] models canonical shapes, CPS uses shape encoding but benefits from ICP refinement, and DualPoseNet [87] and FS-Net [22] add dual decoders or rotation-aware features.Source: §5.2 · p. 23
• Keypoint methods such as MobilePose [56] enable lightweight RGB-only, real-time pose recovery, but produce up-to-scale solutions without absolute camera depth.MobilePose v2 [2] is a two-stage pipeline; Lin et al. [88] later propose a single-stage method that outperforms it, while also remaining up-to-scale.Source: §5.2 · pp. 23–24
• Other 6D approaches include iterative analysis-by-synthesis for RGB-only pose estimation and ANCSH [83] for articulated objects, which predicts per-part poses and joint properties.Analysis-by-synthesis can be sensitive to initialization and slow, while ANCSH uses depth and leaves the use of RGB information unstudied.Source: §5.2 · p. 24
• NOCS evaluations use five joint metrics, while Table 9 reports RGBD results and Table 10 reports MobilePose and MobilePose v2 results on Objectron using mAP@IoU50.The NOCS metrics are mAP@IoU25, mAP@IoU50, mAP@5◦5cm, mAP@10◦2cm, and mAP@10◦5cm.Source: §5.2 · p. 24

6.1 Instance-Level Monocular Object Pose Tracking

• SE(3)-TrackerNet trains only on synthetic data and runs in real time; its reported strengths include rotation-translation disentanglement, Lie Algebra rotation representation, and DR data enhancement.I Like to Move It instead uses reinforcement learning to iteratively update an initial pose through discrete steps.Source: §6.1 · p. 25
• Refinement trackers reuse the previous pose to initialize the current frame, but CAD rendering can complicate end-to-end training and render or model quality may affect results.D6DT and Marougkas et al. use RGBD, while DMB6D uses RGB and aligns image contours; these methods refine with one forward pass, whereas DeepIM iteratively matches rendered and observed images.Source: §6.1 · p. 25
• PoseRBPF combines Rao-Blackwellized particle filtering with a learned autoencoder, but discretizing rotations limits performance and makes complexity grow linearly with the number of rotations.Majcher et al. also use particle filtering, optimizing silhouette and edge differences; optimization-based methods are interpretable but often slow and do not support end-to-end training and testing.Source: §6.1 · pp. 25–26

6.2 Category-Level Monocular Object Pose Tracking

• Category-level tracking predicts 6D poses from a sequence given an initial pose, without recovering object size or using CAD models.The survey excludes CAD-dependent methods such as DMB6D from its category-level classification.Source: §6.2 · p. 26
• Tracking-by-detection methods associate frame-level detections, but separating detection from tracking can cause drift over long sequences.Mono3D-tracking and QD3DT associate vehicle proposals across frames; 3DOT uses a 3D Kalman filter, while MotionNet recovers pitch and roll and performs better in long-range tracking than the compared methods.Source: §6.2 · pp. 26–27
• CenterTrack tracks keypoints end-to-end in a single shot, but estimates only 7Dof object pose.It takes current and previous images plus previous-frame keypoints as input, and predicts current poses and keypoint offsets.Source: §6.2 · p. 27
• Keypoint-based tracking is lightweight and can unify detection and tracking, but some methods provide only up-to-scale pose or depend on depth.6-PACK accumulates relative pose changes using adaptive keypoints and prior poses, but requires RGBD and cannot generalize to RGB-only input.Source: §6.2 · p. 27
• A unified quantitative comparison of tracking methods is difficult because they address different scenarios and use different input formats.The survey also describes planar-target tracking and relative-scale 6Dof tracking without calibration, including a system that runs in real time on mobile phones.Source: §6.2 · p. 27

Instance-level monocular object pose detection

• RGB-only methods are vulnerable to lighting changes and shooting angles, which can obscure image features, especially for keypoint detection.The survey identifies outdoor applications such as mobile AR as particularly challenging because lighting is uncontrollable and unpredictable.Source: §7 · p. 28
• RGBD methods often outperform RGB-only methods but consume more computational resources, and ICP refinement can further increase running time.The survey suggests lightweight architectures and sparse voxel convolution as possible directions to reduce complexity and address slow point-cloud processing.Source: §7 · p. 28
• RGBD methods are evaluated on dense point clouds, while low-power hardware often captures sparse ones, leaving performance in practical use underexplored.The survey calls for evaluating whether current algorithms suit sparse point-cloud input and developing tailored methods if they do not.Source: §7 · pp. 28–29
• Synthetic training data provide ground-truth poses, but models trained on synthetic datasets usually perform poorly on real-world images.The survey identifies improving generalization, including through further self-supervised learning research, as a direction for both RGB-based instance-level and category-level methods.Source: §7 · p. 29

Category-level monocular 3D object detection

• For category-level monocular 3D detection, locating objects from one RGB image is ill-posed, so the survey identifies depth prediction as essential.It proposes studying instance-aware depth perception, including non-local image features, as a possible direction.Source: §7 · p. 29

Category-level monocular 6D pose detection

• Category-level monocular 6D pose methods often use a separate 2D detector before pose prediction, risking accumulated localization errors.The survey raises unified object proposals and pose estimation, or proposal-free methods, as open directions; it also notes large backbones and detection time complicate real-time performance.Source: §7 · pp. 29–30

Monocular object pose tracking

• Tracking methods using only the current and previous frames may underuse sequence information, accumulate errors, and suffer box drift.The survey proposes recurrent models such as LSTMs to associate multiple frames and improve feature use and tracking stability.Source: §7 · p. 30

Datasets and evaluation

• The survey identifies dataset and evaluation gaps, including codemarkers that can bias deployment, limited NOCS instance variation, and no unified tracking benchmark.It also highlights occlusion and truncation, the scarcity of full 9Dof category-level tracking, and the time cost of rendering CAD models as challenges.Source: §7 · p. 31

• The study surveys state-of-the-art deep learning methods for instance-level and category-level monocular pose detection and monocular pose tracking.Source: §8 · p. 31
• It introduces metrics and datasets in detail and presents a comprehensive taxonomy and qualitative and quantitative performance comparisons.Source: §8 · p. 31
• The study analyzes potential research directions and challenges.Source: §8 · p. 31