PaperlayerView exact revision on arXiv

Source-linked AI summary

CosyPose: Consistent multi-view multi-object 6D pose estimation

Yann Labbé, Justin Carpentier, Mathieu Aubry, Josef Sivic

arXiv:2008.08465v1cs.CV

TL;DR

CosyPose addresses multi-object 6D pose estimation from RGB images when camera viewpoints are unknown and single-view hypotheses are noisy or incorrect. It matches pose hypotheses across views, jointly estimates cameras and objects, and globally refines the consistent scene. The method reports state-of-the-art performance on YCB-Video and T-LESS, including a 34.2% absolute improvement on T-LESS for single-view estimation.

  • Problem

    Estimating accurate 6D poses for multiple known objects from multiple RGB views is difficult because camera positions, object visibility, object counts, and reliable correspondences may be unknown.

  • Method

    CosyPose generates single-view pose hypotheses, robustly matches object candidates across views, recovers camera positions, and refines cameras and object poses through object-level bundle adjustment.

  • Results

    CosyPose outperforms state-of-the-art single-view and multi-view methods on YCB-Video and T-LESS, including a 34.2% absolute improvement over on T-LESS.

  • Takeaways & Limitations

    The approach produces a single consistent RGB-only scene reconstruction that handles symmetries, missing or incorrect hypotheses, unknown object counts, and automatically recovered camera poses.

  • Takeaways & Limitations

    The system assumes known 3D object models and known camera intrinsic parameters, while camera relative poses and scene object counts may be unknown.

Abstract

from arXiv · show

We introduce an approach for recovering the 6D pose of multiple known objects in a scene captured by a set of input images with unknown camera viewpoints. First, we present a single-view single-object 6D pose estimation method, which we use to generate 6D object pose hypotheses. Second, we develop a robust method for matching individual 6D object pose hypotheses across different input images in order to jointly estimate camera viewpoints and 6D poses of all objects in a single consistent scene. Our approach explicitly handles object symmetries, does not require depth measurements, is robust to missing or incorrect object hypotheses, and automatically recovers the number of objects in the scene. Third, we develop a method for global scene refinement given multiple object hypotheses and their correspondences across views. This is achieved by solving an object-level bundle adjustment problem that refines the poses of cameras and objects to minimize the reprojection error in all views. We demonstrate that the proposed method, dubbed CosyPose, outperforms current state-of-the-art results for single-view and multi-view 6D object pose estimation by a large margin on two challenging benchmarks: the YCB-Video and T-LESS datasets. Code and pre-trained models are available on the project webpage https://www.di.ens.fr/willow/research/cosypose/.

1 Introduction

CosyPose targets consistent 6D reconstruction of multiple known objects from RGB images taken by cameras with unknown viewpoints. It combines multi-view information to address unreliable single-view hypotheses and produce a unified scene interpretation.

  • Problem: The task is to recover multiple known objects’ 6D poses and camera viewpoints from RGB images captured at unknown camera positions.Challenges include textureless objects, similar instances, unknown object counts and types, and missing visibility across views.
  • Motivation: Single-view RGB methods handle symmetric or textureless objects, but typically estimate object poses independently from one image.CosyPose addresses the resulting limitation by combining multiple views and jointly estimating multiple object poses.
  • Challenges: Unknown camera transformations, false positives, missed detections, and single-view depth ambiguities make cross-view object matching difficult.These issues prevent individual pose hypotheses from being directly expressed in a common reference frame.
  • Approach: CosyPose generates single-view pose hypotheses, matches them across views to recover relative camera positions, and refines the scene with object-level bundle adjustment.The matching procedure uses robust RANSAC-based scene consistency optimization before global refinement.
  • Results: 34.2% absolute improvement over the state-of-the-art on T-LESS is reported for single-view single-object pose estimation.The multi-view framework outperforms on YCB-Video without known camera poses or a single-instance-per-class restriction, and improves over the single-view baseline on both datasets.

2 Related work

Prior work addresses object pose estimation using local features, templates, deep networks, multi-view aggregation, or object-level SLAM. CosyPose differs by recovering camera poses from neural-network-generated 6D hypotheses using RGB images without depth or temporal continuity assumptions.

  • Single-view pose estimation: Feature-based methods struggle with textureless objects, while global template matching can fail under partial occlusion.Deep neural networks have been used to revisit both approaches.
  • Multi-view single-object estimation: Multi-view observations can resolve depth ambiguities and improve robustness to occlusions by matching information across views.Prior methods include feature matching and deep networks that aggregate pose candidates, often assuming known camera poses.
  • Multi-view single-object estimation: CosyPose outperforms despite not assuming known camera poses.Its multi-view formulation requires less information than the compared method.
  • Multi-view multi-object estimation: Depth-based object-level SLAM methods track objects and estimate cameras using depth measurements and temporal continuity between views.CosyPose instead uses RGB images and does not require temporal continuity.
  • Multi-view multi-object estimation: RGB-only object-level SLAM methods rely on local image features, whereas CosyPose uses neural-network 6D pose hypotheses to recover camera poses.CosyPose also handles multiple instances of the same object, unlike methods restricted to one instance per class.

3 Multi-view multi-object 6D object pose estimation

CosyPose reconstructs a consistent multi-object scene from RGB views through candidate generation, cross-view matching, and global refinement. The framework explicitly models symmetries and noisy candidate detections while jointly estimating object and camera poses.

  • 3.1 Approach overview: The system takes multiple photographs and known object models, while allowing repeated object types, missing visibility, unknown object counts, and unknown relative camera poses.Its output is a scene model containing object counts, object 6D poses, and relative camera poses.
  • 3.1 Approach overview: The framework has three stages: generate single-view object candidates, match candidates across views, then globally refine object and camera poses.The final refinement minimizes multi-view reprojection error.
  • 3.2 Stage 1: object candidate generation: Stage 1 produces labeled 6D pose candidates for each detection relative to its image camera, using an RGB single-view estimator that may produce missing or incorrect candidates.The method is compatible with detectors and pose estimators such as FasterRCNN, RetinaNet, PoseCNN, DPOD, and DeepIM.
  • 3.2 Stage 1: object candidate generation: Object symmetries are represented as transformations preserving rendered appearance, and symmetric distance compares poses using the symmetry that best aligns transformed model points.For continuous symmetry axes, the symmetry set is discretized using 64 rotation angles around each axis.
  • 3.3 Stage 2: object candidate matching: For each image pair, RANSAC samples two label-consistent candidate pairs, hypothesizes a relative camera pose, and retains associations with the strongest inlier support.Using two pairs usually resolves symmetry ambiguities; globally, validated pairwise associations form a graph whose isolated vertices are removed.
  • 3.4 Stage 3: scene refinement: 89.8 AUC of ADD-S and 84.5 AUC of ADD(-S) are reported for CosyPose on YCB-Video, exceeding DeepIM’s 88.1 and 81.9.The table compares single-view pose estimation methods on YCB-Video.

4 Results

Experiments on YCB-Video and T-LESS show strong single-view and multi-view performance, with scene refinement improving poses and multi-view estimation recovering cameras robustly without calibration.

  • The evaluation uses YCB-Video and T-LESS benchmarks with multiple views and ground-truth 6D object poses for cluttered scenes.
  • Single-view single-object experiments: 34.2% absolute improvement over existing state-of-the-art methods is achieved on T-LESS for evsd < 0.3 with coarse pose estimation plus refinement.
  • Single-view single-object experiments: Data augmentation is crucial on T-LESS, while the network, loss, and rotation parameterization each provide small but clear improvements.
  • Multi-view experiments: CosyPose significantly outperforms prior work on YCB-Video in both single-view and multi-view scenarios without requiring known camera poses.
  • Multi-view experiments: Using multiple views clearly improves T-LESS results, including the ViVo task evaluated with a variable number of views.
  • Multi-view experiments: Global scene refinement yields around 20% relative improvement in average ADD-S error on both datasets.
  • Multi-view experiments: For groups of 8 T-LESS views, CosyPose recovers camera poses in 74% of cases versus 4% for COLMAP; ground-truth camera poses improve results by at most 3%.
  • Multi-view experiments: A common four-view, six-detection-per-view case takes approximately 320 ms, including pose estimation, candidate association, and scene refinement.

5 Conclusion

CosyPose reconstructs consistent multi-object scenes from several non-calibrated cameras by combining learnable pose estimation, robust matching, and global refinement. The approach handles symmetries and imperfect hypotheses while recovering camera poses and object counts.

  • CosyPose combines learnable 6D pose estimation, robust multi-view matching, and global refinement to reconstruct a single consistent scene.
  • The approach handles object symmetries, requires no depth measurements, tolerates missing or incorrect hypotheses, and recovers camera poses and the number of objects automatically.

A Our single-view single-object method

The single-view method iteratively refines object pose hypotheses from cropped real and synthetic images, using a CNN with tailored pose parametrization, symmetry handling, and training strategies. It achieves strong benchmark performance, including a 34.2% absolute improvement over the state of the art on T-LESS.

  • Pose estimation network: A deep network iteratively refines an object pose using a cropped real image and a synthetic rendering of the object at the current estimate.The network receives concatenated 320×240 crops and predicts translation and rotation updates.
  • Pose parametrization: The network predicts image-space translation, relative depth, and object-independent rotation updates from the current pose.The output comprises [vx, vy, vz] and two vectors used to recover a rotation matrix.
  • Pose parametrization: Relative depth uses a linear parametrization rather than DeepIM’s exponential update, while cropped-camera focal lengths account for crop-induced changes and non-unique T-LESS intrinsics.The authors report that the linear depth parametrization was more stable to train.
  • Loss and symmetry handling: The method addresses symmetric objects by enumerating possible symmetries for the best model-to-ground-truth vertex matching and disentangles translation, depth, and rotation losses.The symmetric distance uses an L1 norm in the described loss.
  • Training and results: Training uses real images plus one million synthetic images per dataset, with randomized multi-object configurations and both flying and physics-simulated placements.On YCB-Video, synthetic pre-training improves AUC of ADD(-S) by approximately 2 points.

D.1 Datasets

The experiments use YCB-Video and T-LESS, which differ in scene composition, object appearance, viewpoint coverage, and evaluation subsets. YCB-Video contains mostly distinct objects, whereas T-LESS emphasizes textureless, symmetric, and visually similar industrial objects.

  • YCB-Video: YCB-Video contains 92 scenes with around 1000 images per scene, split into 80 training and 12 testing scenes.Its challenges include lighting variation, image noise, and occlusions, with 21 reconstructed object models available.
  • YCB-Video: YCB-Video scenes contain at most one instance of each object and mostly visually distinct objects, while evaluation uses 2949 keyframes with limited viewpoint variety.The camera is usually moved in front of the scene rather than fully around it.
  • T-LESS: T-LESS comprises 20 scenes with 30 textureless object instances, most symmetric, and many visually similar enough to challenge object classification.Scene complexity ranges from 3 objects to 18 objects, including up to 7 of the same type.
  • T-LESS: T-LESS images are captured all around scenes, and multi-view experiments use the BOP19 challenge subset while CAD models support training-image generation and evaluation.Single-view experiments use all images from the testing scenes.

D.2 Metrics

The evaluation combines model-based pose-error metrics for symmetric and non-symmetric objects with visibility-aware T-LESS evaluation and a multi-view precision–recall analysis. These choices define correctness using object geometry, symmetry, visibility, and detection matching.

  • Pose metrics: ADD measures average vertex distance for non-symmetric objects, while ADD-S uses closest-point distance for symmetric objects.ADD(-S) denotes using ADD for non-symmetric objects and ADD-S for symmetric objects.
  • Pose metrics: Pose accuracy is reported as the percentage of objects below 10% of object diameter and as AUC over thresholds from 0 to 10 cm.The reported notation includes ADD-S < 0.1d and ADD(-S) < 0.1d.
  • T-LESS metric: On T-LESS, VSD accounts for object visibility and is invariant to object symmetries, with correctness defined by evsd < 0.3 using τ = 20 mm and δ = 15 mm.The official BOP challenge implementation is used.
  • Multi-view evaluation: Multi-view evaluation follows the ViVo BOP top-k protocol, where k equals the number of ground-truth objects of each class in the scene.The protocol selects the highest-scoring predictions for each class in each image.
  • Multi-view evaluation: The authors additionally analyze precision–recall using ADD-S<0.1d true positives because BOP metrics do not penalize extra predictions for absent classes.Scene-refinement mean ADD-S errors retain only predictions below half an object diameter to ensure correct matching.

E Additional multi-view multi-object results

The additional results visualize multi-view scene reconstruction across input images and processing stages, comparing object candidates, reconstructed scenes, rendered viewpoints, and ground truth. The figures are designed to expose inlier selection and consistency across views.

  • Figure layout: Each reconstruction figure uses four T-LESS or five YCB-Video RGB images, with rows corresponding to images and columns to method stages.The final column presents the ground-truth scene.
  • Visualization stages: The visualization stages include input images, detector-based 2D detections, object candidates, scene reconstruction, and ground truth.Detection boxes are filtered below score 0.3 and colored by predicted object label.
  • Candidate filtering: Object candidates marked as red transparent outliers are discarded, while green inliers are retained for the final scene reconstruction.The red and green colors indicate inlier status rather than object correspondence.
  • Scene comparison: After reconstruction, recovered object and camera poses render each scene from predicted viewpoints for overlay against the input images.Ground-truth viewpoints are shown separately for visual comparison, but ground-truth scene information is not used by the method.

E.1 Highlights of the capabilities of our system

CosyPose recovers consistent multi-object scenes under symmetry, occlusion, repeated instances, distractors, and challenging spatial arrangements. Its multi-view refinement supports accurate object poses from only four input images.

  • Robustness to symmetry and occlusion: CosyPose recovers all objects even when scenes contain symmetric objects and severe, view-dependent occlusions.The method correctly estimates eight symmetric objects although some are missing from individual views.
  • Multiple object instances: The method identifies the correct number and labels of repeated object instances despite similar appearance and partial occlusion.This capability is demonstrated for multiple objects of the same type across views.
  • Distractor robustness: CosyPose filters database-external distractors while recovering all database objects in cluttered scenes.This behavior is especially relevant when known objects must be distinguished from background objects in unstructured environments.
  • Accuracy: Scene refinement significantly improves pose accuracy by using information from multiple views, with accurate reconstructions demonstrated from four input images.Rendered novel viewpoints show precise object contacts and insertion relationships.

E.2 Detailed examples

Detailed examples show how CosyPose uses cross-view consistency to recover partially occluded objects, reject incorrect hypotheses, suppress duplicates, and filter distractors. Novel-view renderings also illustrate accurate spatial relationships in the reconstructed scene.

  • Robustness to missing detections: Cross-view matching recovers a partially or fully occluded object's pose when it is visible elsewhere and the cameras can be positioned from other objects.The method can position both the affected view and the occluded object using complementary observations.
  • Robustness to incorrect detections: For conflicting labels at one image location, CosyPose generates hypotheses for each label and discards candidates that cannot match consistently across views.This handles incorrect detections caused by visually similar objects in T-LESS.
  • Duplicate objects: Consistent alternative labels for the same physical object can produce duplicate 3D objects, which CosyPose removes with 3D non-maximum suppression.When objects are too close, the system keeps the candidate with the highest summed 2D detection score.
  • Accuracy: Novel-view renderings show accurate contact and insertion geometry, including a yellow object touching green objects and a brown object plugged inside it.These relationships are evaluated from viewpoints different from those used for reconstruction.
  • Distractors and false positives: False-positive distractor detections are filtered when their estimated poses are inconsistent across views.Such inconsistencies commonly arise because the pose estimator was not trained on those distractor objects.

E.3 Limitations

The reported failure cases arise when incorrect hypotheses are mutually consistent, correct detections lack cross-view matches, or too few reliable object inliers support camera estimation.

  • Consistent mistakes: Consistent incorrect candidates across at least two similar views can create an incorrect object in the reconstructed scene.More views or requiring a sufficiently large number of associated candidates could address this failure mode.
  • Missing objects: A correct candidate detected in only one view is omitted because the current approach requires a match in another view.In this case, camera poses can still be estimated correctly; guided matching is suggested as a possible remedy.
  • Incorrect camera pose estimates: Camera pose estimation requires at least three object-candidate inliers, so views with only two visible objects or incorrect detections may be mispositioned.Insufficient inliers prevent correct reprojection of objects in the affected image.
Loading 2008.08465v1…