Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors

• 3D cuboids capture location, orientation, and metric extent, but obtaining enough labels is costly and existing semiautomatic workflows still need human correction on difficult cases.Source: §1 · p. 1
• The pipeline combines NOCS with DeepSDF in Coordinate Shape Space (CSS) for shape correspondence, introduces a differentiable SDF renderer, and uses curriculum-based autolabeling.Source: §1 · p. 2
• Off-the-shelf 2D detections seed CSS predictions; NOCS coordinates are projected into LIDAR, matched to a decoded model, then refined through projective and geometric alignment.Source: Figure 1 · p. 1, §1 · p. 2
• The pipeline retrains CSS on recovered labels over successive rounds, beginning with easy samples and increasing difficulty to limit drift from noisy labels.Source: §1 · p. 2
• On KITTI3D, the method recovers metric cuboids with structural differentiable priors, and the labels can train efficient 3D object detectors.Source: §1 · p. 2

• Assisted-labeling systems use 2D detectors, predictive models, or LIDAR to support human annotation, whereas this work investigates automatic labeling with geometric verification [2] [25] [22] [10].Source: §2 · p. 2
• Differentiable rendering work covers mesh rasterization [15] [4], soft rasterization [26], and path tracing.Source: §2 · p. 2
• Car shape-space methods [29] lack end-to-end differentiability or approximate gradients with finite differences; DeepSDF enables backpropagation over a smooth manifold and avoids mesh-topology problems [30].Source: §2 · p. 2
• DensePose maps canonical coordinates to human bodies, and later work uses them with differentiable rendering for pose, shape, texture, or motion [18] [19]; these methods recover projective scenes only up to scale.Source: §2 · p. 2
• Prior dense-coordinate methods address monocular pose for known CAD models [31] [17] [33], while NOCS recovers metric scale and pose from RGB coordinates plus depth [38].Source: §2 · p. 2
• Related SDF-prior work uses stereo depth or temporal priors for pose and shape [8], while [37] assumes correct localization and focuses on reconstruction quality.Source: §2 · p. 3

3.1. Coordinate Shape Space

• CSS combines DeepSDF car-shape representations with NOCS coordinates, using unit-diameter models to support approximate deformable shape matching.DeepSDF represents models as signed distance fields in a compact latent shape space.Source: §3.1 · p. 3

3.2. Differentiable SDF Rendering

• SDF gradients provide surface normals, which combine with signed distances to project query locations onto differentiable surface points.Queries outside the narrow band |s| ≤ 0.03 are discarded; the resulting points support derivatives with respect to scale, pose, and latent code.Source: §3.2 · p. 3
• Oriented tangent discs cover projected surface samples, whose depth-weighted softmax contributions form differentiable NOCS renderings.Disc diameters are chosen to close holes; the rendering weights make contributions sum to one at each pixel and permit gradient flow from pixels to surface points.Source: §3.2 · p. 4

3.3. 3D Autolabeling Pipeline

• A CSS network trained on synthetic patches predicts NOCS maps and shape vectors, while the curriculum admits real annotations from easier to harder instances.Difficulty depends on pixel size, overlap with other 2D labels, and image-border contact.Source: §3.3 · pp. 4–5

3.3.1 Initialization and Optimization

• Initialization matches decoded model-surface points to LIDAR points in NOCS space, then uses Procrustes with RANSAC to estimate pose and scale.The method retains nearest neighbors whose NOCS-coordinate distance is below 0.2.Source: §3.3.1 · p. 5
• Differentiable optimization combines projective NOCS correspondence loss with a 3D LIDAR distance loss to refine shape, pose, and scale.The 2D loss matches nearby NOCS coordinates; the 3D loss keeps model-to-LIDAR neighbors within 0.25 m.Source: §3.3.1 · p. 5

3.3.2 Verification and CSS Retraining

• Verification rejects autolabels failing either the LIDAR surface-band check or the rendered-mask overlap threshold before adding survivors to the training pool.The checks require at least 60% of LIDAR points outside a 0.2 m surface band and mask IoU of at least 70%.Source: §3.3.2 · p. 5
• Surviving real autolabels join synthetic samples for retraining, and successive self-improving loops yield better CSS predictions, initializations, and autolabels.The label pool grows across loops as verified samples are added.Source: §3.3.2 · p. 5

4. Experimental Evaluation

• The KITTI3D evaluation covers 7481 car frames and relaxes the usual 3D IoU threshold because ground-truth cuboids vary in padding while autolabels fit surfaces tightly.BEV and 3D IoUs are evaluated at 0.5, while NS uses 0.5 m and 1 m distance cutoffs.Source: §4 · p. 6

4.1. Correctness of autolabels

• The study evaluates both KITTI3D ground-truth 2D boxes and off-the-shelf Mask-RCNN boxes or masks, with easy and moderate cases defined by image size and overlap.The detector-based scenarios use a COCO-trained ResNeXt101 Mask-RCNN.Source: §4.1 · pp. 6–7, Table 1 · p. 7
• For easy samples, autolabels achieve approximately 80% BEV AP and 60% 3D AP; moderate samples reach approximately 60% BEV AP after retraining.The estimated NS scores indicate that more than 90% of autolabels are within one meter of the real location.Source: §4.1 · p. 7, Table 1 · p. 7
• Harder samples show an approximately 20-point drop across metrics, as occlusion and distance can make RANSAC correspondence finding difficult and reduce recall.The method requires a minimum set of 3D-3D RANSAC inliers for proper solutions.Source: §4.1 · p. 7
• NOCS predictions become more accurate over successive loops, and two loops suffice to stabilize the results.The initial predictions are noisy because of the synthetic domain gap.Source: §4.1 · p. 7, Figure 8 · p. 7

4.2. Ablation

• Pose optimization raises 3D IoU from 41.85% to 63.42%, while the RANSAC baseline has NS scores of 81.36% and 95.45%.The authors interpret the IoU improvement, alongside similar NS scores, as evidence of substantially better rotations.Source: §4.2 · p. 7, Table 3 · p. 8
• Combining 2D and 3D losses trades BEV AP (80.61 to 78.09) for 3D AP (60.92 to 63.53).Optimizing only in 2D causes a drastic drop in 3D metrics because scale and translation can drift.Source: §4.2 · p. 7, Table 3 · p. 8

4.3. 3D Object Detection

• Autolabel-trained detectors achieve BEV performance competitive with detectors trained on true KITTI labels at IoU 0.5 and 0.7, while their 3D AP at IoU 0.5 reaches 70−90% of true-label performance.At IoU 0.7, 3D performance worsens because KITTI labels are often inflated relative to tight autolabels.Source: §4.3 · p. 8

5. Conclusion

• The pipeline bootstraps parametric 3D instance recovery from synthetic data and off-the-shelf detectors, then improves through self-improving autolabeling.Its core combines dense surface coordinates, a shape space, and differentiable SDF rendering.Source: §5 · p. 8
• The approach recovers a substantial amount of high-precision cuboid labels that can train 3D object detectors with results close to state of the art.Source: §5 · p. 8
• Future work will investigate parametric reconstruction of additional categories, including pedestrians and road surfaces.Source: §5 · p. 8

A. DeepSDF

• The DeepSDF network is trained to cover normalized predefined cars from the Parallel Domain dataset.Source: A. DeepSDF · p. 11
• The latent shape vectors have dimensionality three, allowing their positions to be visualized on the surface of a 3D sphere.Source: A. DeepSDF · p. 11
• Each red point in the sphere visualization represents a model shape embedded in the latent space.Source: A. DeepSDF · p. 11

B. Data Generation

• Ground-truth poses and CAD models from the synthetic Parallel Domain dataset are projected using DeepSDF and the differentiable renderer.Source: B. Data Generation · p. 11
• The rendered outputs include normalized object coordinates as RGB channels and object normals used for 0-isosurface projection.Source: B. Data Generation · p. 11
• Training augmentations alter crops with rotations, flips, cropping, and color changes, while normal maps support Phong-shaded ambient, diffusive, and specular lighting.Source: B. Data Generation · p. 11

C. Pipeline Components

• The PD cars train the DeepSDF shape space, while a separate visualization pairs latent vectors with shapes after 0-isosurface projection.Source: Figure 10 · p. 11, Figure 11 · p. 11

C.1. CSS Net

• The CSS network uses a ResNet18 backbone and bilinear-interpolation decoders to reduce parameters and computation relative to deconvolution.Each upsampling stage concatenates the previous-level feature map and applies a convolution.Source: §C.1 · p. 11
• ImageNet initialization and freezing the first five layers address overfitting to rendered synthetic data; five heads predict NOCS channels, a mask, and a DeepSDF latent vector.Source: §C.1 · p. 11

C.2. Pose Estimation Block

• The CSS network maps RGB pixels to surface NOCS, which are backprojected onto LIDAR frustum points using camera parameters.Source: §C.2 · p. 11
• The latent vector decodes to a normalized surface point cloud through 0-isosurface projection, whose model points represent NOCS.Source: §C.2 · p. 12
• Nearest-neighbor NOCS correspondences are filtered with RANSAC, and Procrustes on four sampled matches estimates initial pose and scale.Source: §C.2 · p. 12
• RANSAC uses a 0.2 m inlier threshold, with success probability p = 0.9 and inlier probability w = 0.7.Source: §C.2 · p. 12

C.3. Optimization

• Each iteration compares the transformed, back-face-culled surface cloud with LIDAR points and rendered NOCS with CSS predictions, then updates pose and latent code.The pipeline figure depicts the CSS outputs, surface projection, pose transform, 3D losses, and dense screen-space rendering losses.Source: §C.3 · p. 12, Figure 14 · p. 13
• 3D losses align shape and pose to frustum points but can be poorly constrained by sparse data; dense 2D losses aid screen-space alignment but depend on initial estimates for scale and translation.Source: §C.3 · pp. 12–13

D. Surface Tangent Discs

• The tangent-disc construction solves for each screen pixel’s distance to a plane using perspective-projection and plane equations.Source: D. Surface Tangent Discs · p. 13
• The resulting 3D plane position for each screen pixel supports forming primitives from distances to 3D shape points.Source: D. Surface Tangent Discs · p. 13