WAPR: A Foundation Model for Wide-Angle Refinement in Unseen Object Pose Estimation

1 Introduction

• Sparse candidates reduce runtime but can leave larger angular deviations for refinement, while object-specific training limits deployment scalability.Large deviations can arise from coarse hypotheses, occlusion, and viewpoint changes.Source: §1 · p. 2
• WAPR compares rendered and real RGB-D observations and regresses rotation and translation updates after training on pose perturbations up to 90°.Its wide-angle refinement supports accurate pose estimation with as few as 12 candidates in the fast setting.Source: §1 · p. 2
• Symmetry-equivalent candidate–target poses can receive inconsistent raw rotation targets, so WAPR canonicalizes pose targets before loss computation.The canonicalization makes wide-angle candidate–target pairs well-defined for rotationally symmetric objects.Source: §1 · pp. 2–3
• SA6D adds KASAL-assisted symmetry priors for 944 GSO scans and expands them into about 50K augmented instances and 2M synthetic RGB-D images; angle-balanced loss reduces large-error samples’ dominance.The loss is intended to stabilize wide-angle optimization across angular ranges.Source: §1 · p. 3
• 84.5% AR / 85.3% AP in unconstrained Multi-Det. improves over FreeZeV2 by 1.2% AR / 2.0% AP; fast MUSE reaches 80.6% AR / 78.6% AP.With Co-op’s F3DT2D detector, WAPR obtains 78.1% AR / 76.5% AP, a matched-detector gain of +2.2% AR / +7.2% AP.Source: §1 · p. 3

2.1 6D Pose Estimation for Seen Objects

• Seen-object pipelines progress from direct pose regression to keypoint or dense-coordinate prediction with PnP, while refinement aligns rendered observations with real images.These object-specific pipelines provide a basis for unseen-object methods, but their training requirements motivate reducing object-specific training.Source: §2.1 · pp. 3–4

2.2 6D Pose Estimation for Unseen Objects

• Unseen-object methods use object models with minimal or no retraining, and candidate-based pipelines trade pose coverage against inference runtime and refinement demands.WAPR targets sparse candidates with larger residual deviations than many existing refiners handle effectively.Source: §2.2 · p. 4

2.3 Symmetry Handling in 6D Pose Learning and Evaluation

• Symmetry-aware metrics and learning objectives treat visually equivalent poses consistently; SA6D extends this principle to dataset-level priors for canonicalizing synthetic training pairs.Prior synthetic datasets emphasized object diversity and scalable pose supervision, while SA6D adds KASAL-assisted rotational symmetry priors at scale.Source: §2.3 · pp. 4–5

3 Methodology

• WAPR predicts rotation and translation updates from paired real and rendered RGB-D observations and masks, while angle-balanced loss stabilizes training across pose deviations.Sparse candidate generation supports inference with 12 or 24 candidates.Source: §3 · pp. 5–6

3.1 Symmetry-Aware 6D Dataset (SA6D)

• SA6D starts from 944 GSO scans, obtains KASAL-assisted rotational symmetry priors, and expands them into about 50K augmented instances and about 2M synthetic RGB-D images.About 30K of the instances are rotationally symmetric; BlenderProc renders the training images.Source: §3.1 · p. 6, Figure 3 · p. 6

Object Collection and Symmetry Augmentation

• Geometry scaling and texture randomization reuse each scan’s priors, while uniform-color replacement removes texture-induced asymmetry and uses the geometry-only prior.The other texture augmentation preserves the original texture-aware rotational symmetry prior.Source: Object Collection and Symmetry… · p. 6

Symmetry Disambiguation

• Offline min-over-symmetry canonicalization selects the variant with the smallest rotation from identity, yielding a unique update target independent of network predictions.The target pose is canonicalized under the same prior to make the candidate–target pair consistent.Source: Symmetry Disambiguation · pp. 7–8
• Random rotation and translation perturbations of ground-truth poses generate the augmented candidate poses used for training.The set is denoted P_aug in the passage.Source: Symmetry Disambiguation · p. 7
• An object’s symmetry center is aligned to the origin, and its rotational symmetry is represented by discrete rotations or a discretized continuous orbit.The continuous orbit uses 360 uniformly spaced rotations, bounding angular quantization error by 0.5°.Source: Symmetry Disambiguation · p. 7
• For each perturbed pose, the method constructs symmetry-equivalent variants by composing its rotation with each valid symmetry rotation while retaining its translation.These variants form the candidate set from which the canonical representative is selected.Source: Symmetry Disambiguation · p. 7

Segmentation Mask Generation

• SAM generates candidate training masks, avoiding the cleaner perfect synthetic masks that would differ from inference-time inputs.The masks are generated with a large-scale segmentation model such as SAM.Source: Segmentation Mask Generation · p. 8
• Candidates with IoU above 0.5 against the ground-truth mask are retained; one is sampled, or a blank mask is used if none qualify.Source: Segmentation Mask Generation · p. 8

Neural Network

• Paired observed and candidate-rendered RGB-D crops and masks feed shared-weight feature extractors, followed by fused Transformer features and separate rotation and translation update heads.Training uses PBR-rendered observations; inference uses real sensor observations.Source: Neural Network · p. 8
• Rotation updates are represented as 3D rotation vectors and converted to SO(3) matrices before application to candidate poses.Source: Neural Network · p. 8
• Training perturbs poses by up to 90° and iteratively applies predicted updates, rerendering inputs after each round; this range is described as wide yet learnable.Larger deviations, especially near 180°, have very low visual correspondence and make convergence less stable.Source: Neural Network · p. 9

Angle-Balanced Loss

• High-error pairs with limited view overlap can dominate gradients despite providing little useful correspondence, motivating an angle-balanced loss.Source: Angle-Balanced Loss · p. 9
• Samples with rotation-vector residuals below τ = 0.25π receive larger dynamic weights, while cases above τ are down-weighted; σ is set to 3.The residual is the L1 magnitude between predicted and target rotation-update vectors.Source: Angle-Balanced Loss · p. 9
• The final objective combines weighted rotation-update L1 error with translation L1 error, using α = 3 and β = 1.Source: Angle-Balanced Loss · p. 9

Inference Pipeline

• The fast setting forms 12 candidates from four tetrahedron-vertex viewing directions and three in-plane rotations; the unconstrained setting forms 24 from eight cube-vertex directions and the same rotations.Source: Inference Pipeline · p. 10
• All 24 unconstrained candidates receive five refinement rounds, whereas the fast setting refines 12 candidates for two rounds before continuing with the top four for three more.Each round uses rerendered inputs; candidates are paired with observed and rendered RGB-D crops and masks.Source: Inference Pipeline · p. 10
• The final pose is selected from the refined candidates using the highest confidence score predicted by FoundationPose’s ScoreNet module.Source: Inference Pipeline · p. 10

Datasets & Metrics

• The evaluation follows the BOP 2024 unseen-object protocol on seven core datasets, reporting AR for localization and AP for detection; the fast setting reaches 25 detected instances per second.WAPR is trained for 800K iterations, with candidate rotations perturbed up to 90° and five-step iterative refinement.Source: Datasets & Metrics · pp. 10–11, Network Settings · p. 10, Fast inference · p. 11

Number of Candidate Poses

• On IC-BIN, mean AP rises from 56.9% with 4 candidates to 70.1% with 12, while larger candidate sets provide smaller gains.The study uses 12 candidates for fast inference and 24 for greater coverage when inference time is unconstrained.Source: Number of Candidate Poses · pp. 11–12

Angle Range for Training

• Training with ±90° rotations performs best overall: ±20° provides insufficient wide-angle coverage, while ±180° adds low-correspondence samples and destabilizes optimization.Removing masks slightly reduces performance, while perfect masks cause a larger drop because they differ from predicted inference masks.Source: Angle Range for Training · pp. 12–13
• Symmetry priors improve AP by 4.2% without angle-balanced loss and 1.7% with it; SA6D training improves AP by 5.7% and 5.6% over matched GSO-PBR counterparts.The comparisons use controlled model configurations.Source: Angle Range for Training · p. 13
• Angle-balanced loss improves AP by 6.1% over training without that loss or symmetry priors, and the combination with symmetry priors improves AP by 7.8%.Source: Angle Range for Training · p. 13
• Replacing attention pooling with average pooling reduces AP by 0.9%, and also removing the Transformer encoder produces a total 1.5% drop from A0.The results indicate that global feature interaction and attention-based aggregation are useful for wide-angle refinement.Source: Angle Range for Training · p. 13
• As a plug-in refiner, mask-free WAPR improves FreeZeV2 AP by 0.6% and GigaPose+GenFlow AP by 1.3%; against refinement baselines, it improves mean AR over OSOP+ICP by 29.7% and AR over PPF/SIFT+Zephyr by 26.7%.WAPR is 4× faster than OSOP+ICP; replacing WAPR with ICP under the same hypothesis setting reduces AR by 54.2%.Source: Angle Range for Training · p. 13

4.3 BOP Results on 6D Localization and Detection

• On seven BOP datasets, fast WAPR reaches 80.6% AR and 78.6% AP, improving over Co-op by 4.7% AR and 9.3% AP; unconstrained WAPR reaches 84.5% AR and 85.3% AP, improving over FreeZeV2 by 1.2% AR and 2.0% AP.The fast setting is evaluated under a < 1 s per-image budget.Source: Table 4 · p. 14, §4.3 · pp. 14–15

• Across seven BOP core datasets, WAPR achieves strong performance in both fast and unconstrained settings.Source: §5 · p. 15
• WAPR refines sparse candidate poses by comparing real and rendered RGB-D observations with masks, handling large candidate-pose deviations.Source: §5 · p. 15
• SA6D provides large-scale synthetic RGB-D training data with KASAL-assisted rotational symmetry priors.Source: §5 · p. 15
• Symmetry priors canonicalize equivalent candidate–target pose pairs, while angle-balanced loss stabilizes learning across angular ranges.Source: §5 · p. 15