Source-linked AI summary
μ-MAR: Multiplane 3D Marker based Registration for Depth-sensing Cameras
Marcelo Saval-Calvo, Jorge Azorin-Lopez, Andres Fuster-Guillo, Higinio Mora-Mora
TL;DR
Multiple-view registration is difficult for low-SNR, low-resolution depth data, especially when transformations between views are unknown. μ-MAR uses iteratively registered multiplane 3D markers to align views and transfer their transformations to static objects. Experiments report high registration accuracy on synthetic and real RGB-D data, while the authors identify extension to general scenes as future work.
Problem
Unknown inter-view transformations and noisy, low-resolution depth data limit reliable multiple-view registration for detailed reconstruction.
Method
μ-MAR registers multiplane 3D markers across multiple views, then applies the marker transformations to the static object for reconstruction.
Results
Hausdorff distance was 0.6556 for μ-MAR versus 6.1515 for ICP, while another evaluation reports average error 0.8958 for μ-MAR versus 8.1154 for ICP.
Takeaways & Limitations
The proposed marker-based approach provides high-accuracy registration for noisy depth data in the evaluated object-reconstruction setting.
Takeaways & Limitations
The authors identify extending the proposal to general scenes and enabling the registration method to decide by itself as future work.
Abstract
from arXiv · showhide
Many applications including object reconstruction, robot guidance, and scene mapping require the registration of multiple views from a scene to generate a complete geometric and appearance model of it. In real situations, transformations between views are unknown an it is necessary to apply expert inference to estimate them. In the last few years, the emergence of low-cost depth-sensing cameras has strengthened the research on this topic, motivating a plethora of new applications. Although they have enough resolution and accuracy for many applications, some situations may not be solved with general state-of-the-art registration methods due to the Signal-to-Noise ratio (SNR) and the resolution of the data provided. The problem of working with low SNR data, in general terms, may appear in any 3D system, then it is necessary to propose novel solutions in this aspect. In this paper, we propose a method, μ-MAR, able to both coarse and fine register sets of 3D points provided by low-cost depth-sensing cameras, despite it is not restricted to these sensors, into a common coordinate system. The method is able to overcome the noisy data problem by means of using a model-based solution of multiplane registration. Specifically, it iteratively registers 3D markers composed by multiple planes extracted from points of multiple views of the scene. As the markers and the object of interest are static in the scenario, the transformations obtained for the markers are applied to the object in order to reconstruct it. Experiments have been performed using synthetic and real data. The synthetic data allows a qualitative and quantitative evaluation by means of visual inspection and Hausdorff distance respectively. The real data experiments show the performance of the proposal using data acquired by a Primesense Carmine RGB-D sensor. The method has been compared to several state-of-the-art methods. The ...
1. Introduction
Multiple-view registration is critical for reconstructing complete 3D models, but low-cost RGB-D sensors provide noisy, low-resolution data that challenges standard methods. μ-MAR addresses this problem with model-based multiplane registration using 3D markers.
- Problem: Registering multiple views into a common coordinate system is necessary because viewpoints and inter-view transformations are generally unknown.The registration quality directly affects the accuracy of the reconstructed model.
- Motivation: Low-cost RGB-D cameras provide limited resolution and substantial depth error, hindering detailed reconstruction of objects and scenes.The Kinect example provides a 320x240 depth matrix and insufficient detail for small shape features.
- Existing methods: ICP provides fine two-cloud registration but requires a suitable initial transformation and can converge to a local minimum.RANSAC is often used to estimate ICP’s initial transformation.
- Existing methods: RANSAC-based and related registration methods tolerate noise only up to a certain level, leaving low-SNR RGB-D data challenging.Prior plane-based methods may also require many planes or large planar regions for reliable estimation.
- Contribution: μ-MAR uses 3D markers composed of multiple planes to obtain coarse and fine transformations for registering views and reconstructing objects.The method applies marker transformations to the static object and is not restricted to RGB-D sensors.
2. Overview of the Multiplane 3D Marker based Registration method
μ-MAR places known multiplane markers around an object, registers those markers across many nearby views, and applies the resulting transformations to the object. Plane models reduce noise effects, but marker number and placement affect accuracy and processing time.
- Robustness: Plane-based marker models reduce noise effects and enable proper alignment when state-of-the-art methods are unsuitable for the data quality.The method assumes many nearby views to capture object parts and reduce occlusion effects.
- Method overview: The method reconstructs an object by placing known multiplane markers around it and acquiring multiple views with a depth-sensing camera.Examples of markers include cubes and pyramids.
- Method overview: μ-MAR has three main stages: plane detection and model extraction, marker-model registration, and object registration.The object receives the same rotations and translations calculated for the markers.
- Marker registration: The marker-registration stage is the core of μ-MAR and uses geometric marker models in an iterative multiview registration process.This stage provides fine registration of planar models.
- Practical constraints: Marker count trades accuracy against processing time, while at least three non-coplanar visible planes are required to avoid undesired registration results.When few planes are available, ICP may be applied within each multiview iteration with strong match rejection.
3. Plane detection and marker model extraction
The marker model-extraction stage detects planar regions from segmented marker data and fits constrained geometric plane models. The resulting models represent each marker’s visible faces using normals and centroids.
- Marker flexibility: Marker shapes and locations can vary because the extraction procedure does not require a fixed size or position.Cubes and pyramids are given as example marker types.
- Plane representation: A plane model is represented by a normal vector and a point, with the point chosen as the centroid of the relevant planar region.This representation targets the specific marker-face areas used for registration.
- Model estimation: The approach extends RANSAC with prior marker geometry to estimate planar models accurately from noisy segmented points.The prior knowledge consists of constraints on the marker’s geometric features.
15 end
The model-estimation algorithm clusters marker points, fits candidate planes, checks marker constraints, and expands accepted models with additional inlier points. It returns planar models defined by normals and centroids.
- Output: The procedure returns a set of planar models whose normals and centroids fit the marker’s input 3D points.The cluster structure is built dynamically while processing the marker data.
- Initialization: The algorithm first clusters the marker point cloud into m regions using point and normal information from k-means.For a cube, m is based on the maximum visible faces plus an allowance for noise.
- Candidate fitting: RANSAC estimates plane models from subsets of each cluster before testing whether the candidate planes satisfy the marker’s geometric constraints.The estimated models include plane centroids and normals.
- Validation: After a candidate passes the constraint test, additional points are evaluated as inliers and the constraints are checked again before accepting the result.This final validation supports the returned planar model set.
4. Registration of marker models
μ-MAR registers multiple views by matching planar marker models and iteratively aligning subsets of views. It handles changing visibility and propagates transformations across consecutive subsets to maintain scene coherence.
- μ-MAR uses previously estimated planar models to register external markers and extends this approach to multiple planar objects in the scene.The method uses plane models represented by normals and centroids.
- The multi-view process iteratively treats one view as Data and the remaining views, or their model, as the target Scene.The Scene may be a concatenation, mean, spline, voxelization, or other model of the views.
- For large view sets, μ-MAR registers consecutive subsets using correspondence estimation, rotation and translation finding, and complete scene adjustment.After each subset is registered, a new subset is selected until all captured views are registered.
- Each view is aligned using marker-plane normals and centroids against the mean corresponding plane models from the other views in its subset.The process repeats until all views in the subset are registered, then transformations are propagated to neighboring subsets.
- When views are widely separated, an initial pairwise registration is suggested because averaging them can produce incorrect normals and centroids.The method assumes that a 3D point cloud and estimable planes are available, and it can use any sensor meeting those conditions.
- 4.1. Correspondence estimation: Dynamic plane correspondences accommodate planes that disappear or appear across viewpoints, while comparisons with recent views improve robustness.The structure uses one column per view and rows for planes, with empty entries when planes are not visible.
5. Experimentation
The experiments evaluate μ-MAR with synthetic and real RGB-D data, comparing its registration performance against ICP and other state-of-the-art methods. Synthetic tests use visual inspection and Hausdorff distance, while real tests assess reconstructed objects qualitatively.
- Synthetic data: Synthetic experiments simulate a 360-degree turntable with four cube markers and target objects using Blender and Blensor.The setup includes 60 captures separated by 6 degrees and tests cube, pyramid, and double-pyramid targets.
- Real data: Real experiments use a Primesense Carmine RGB-D camera, a controlled turntable, 64 captures, and visual comparison with six registration methods.The objects are acquired over 360 degrees with 5.625 degrees between frames.
- Synthetic data: μ-MAR properly registers cubes across noise levels, while ICP slides laterally when only frontal and top planes are registered.Pyramids show a similar geometry-related ICP problem, whereas μ-MAR provides highly accurate registration.
- Synthetic data: With more varied double-pyramid geometry, ICP performs similarly to μ-MAR in the reported visual comparison.The double pyramid contains more planes and more detailed geometry than the cube and pyramid cases.
- Synthetic data: Table 3 reports better μ-MAR results for all tested situations using mean and RMS Hausdorff measures.Min and Max are outlier-dependent, so the study emphasizes global mean and RMS values.
- Real data: μ-MAR accurately registers the real objects and preserves more points than methods using down-sampling during final refinement.The Tool retains more detail than RGBDemo, and the Bomb reconstruction includes the toy’s feet.
6. Discussion
The discussion frames μ-MAR as a model-based multiplane registration method for noisy, low-resolution RGB-D data. It reports improved synthetic and real-data registration, while identifying calibration, marker visibility, and sensor resolution as practical boundaries.
- Motivation: Low-cost RGB-D sensors can provide noisy, low-resolution data that makes small shape features difficult to perceive and match.Depth noise is reported at approximately millimetre orders, with a low Signal-to-Noise ratio.
- Method: μ-MAR registers multiple marker planes simultaneously, using prior marker knowledge to infer view correspondences and transformations.The marker-derived transformations are applied to the object being reconstructed.
- Method: Because registration focuses on external markers rather than object characteristics, μ-MAR can register objects independently of their colour, surface optics, or shape.Proper marker segmentation and planar models for each marker face are required.
- Practical considerations: Default sensor calibration can produce incorrect data near image extremes, while calibration improves registration accuracy.Without calibration, the authors recommend placing the camera and markers away from image extremes.
- Limitations: At least one external marker must remain visible to the camera for object reconstruction.This is identified as the method’s main drawback.
- Results: Synthetic experiments report mean Hausdorff errors of 0.6556 for μ-MAR and 6.1515 for ICP, and RMS errors of 0.8958 and 8.1154, respectively.The authors conclude that μ-MAR is close to 10 times better in average than ICP.
- Results: Real-data experiments with a general-purpose RGB-D sensor validate the proposal, and visual inspection reports μ-MAR outperforming the other tested methods.The comparison includes ICP, RANSAC-based methods, KinectFusion, and RGBDemo.
7. Conclusions
μ-MAR registers multiple views into a common coordinate system using planar 3D markers and iterative multi-view refinement, while minimizing noise effects in RGB-D data. The method supports accurate alignment and transfers marker transformations to reconstruct static objects, with broader scene extensions planned.
- 7. Conclusions: μ-MAR transforms multiple views into a common coordinate system while minimizing noise effects common in RGB-D cameras.The method uses planar models as 3D markers in a model-based registration framework.
- 7. Conclusions: The method simultaneously registers planes using normals for rotation estimation and centroids for translation finding.Planes serve as geometric models and 3D markers for the registration process.
- 7. Conclusions: μ-MAR provides fine, highly accurate view alignment without previous coarse registration.Its multi-view registration variant operates iteratively on subsets of views.
- 7. Conclusions: The method applies the same transformations to the whole scene, enabling high-accuracy reconstruction of static objects.Marker registration supplies the transformations used to align and reconstruct the object.
- 7. Conclusions: Future work extends μ-MAR to general scenes containing planar objects, mixed marker shapes, and viewpoints above or below the object.A double pyramid marker is proposed as a possible way to improve recovery from self-occlusions.
- 7. Conclusions: Future extensions aim to use prior knowledge about reconstruction accuracy and shape complexity to select viewpoints needed for reconstruction.The stated objective is for the registration method to decide which viewpoints should be captured for the required accuracy.