Source-linked AI summary
Interacted Planes Reveal 3D Line Mapping
Zeran Ke, Bin Tan, Gui-Song Xia, Yujun Shen, Nan Xue
TL;DR
3D line mapping must recover reliable line structures from multi-view images despite ambiguous correspondences and imperfect depth, while preserving scene topology. LiP-Map jointly optimizes learnable planes and line segments through their geometric interaction, achieving accurate line mapping and stronger line-assisted localization across indoor-scene benchmarks.
Problem
3D line mapping remains difficult because parallax and inconsistent depth maps make line placement unreliable, while true lines are tied to structural surface edges.
Method
LiP-Map jointly optimizes learnable 3D planar primitives and line segments using multi-view 2D lines together with depth and normal representations.
Results
LiP-Map improves accuracy and completeness over state-of-the-art methods across more than 100 scenes and substantially improves line-assisted visual localization on 7Scenes.
Takeaways & Limitations
Explicit plane–line interaction provides a structured route to accurate, detailed 3D line mapping while also supporting planar reconstruction and visual localization.
Takeaways & Limitations
Neural-field-based matching-free methods can suffer from slow optimization, while LiP-Map relies on depth and normal maps that may be ground-truth or predicted.
Abstract
from arXiv · showhide
3D line mapping from multi-view RGB images provides a compact and structured visual representation of scenes. We study the problem from a physical and topological perspective: a 3D line most naturally emerges as the edge of a finite 3D planar patch. We present LiP-Map, a line-plane joint optimization framework that explicitly models learnable line and planar primitives. This coupling enables accurate and detailed 3D line mapping while maintaining strong efficiency (typically completing a reconstruction in 3 to 5 minutes per scene). LiP-Map pioneers the integration of planar topology into 3D line mapping, not by imposing pairwise coplanarity constraints but by explicitly constructing interactions between plane and line primitives, thus offering a principled route toward structured reconstruction in man-made environments. On more than 100 scenes from ScanNetV2, ScanNet++, Hypersim, 7Scenes, and Tanks\&Temple, LiP-Map improves both accuracy and completeness over state-of-the-art methods. Beyond line mapping quality, LiP-Map significantly advances line-assisted visual localization, establishing strong performance on 7Scenes. Our code is released at https://github.com/calmke/LiPMAP for reproducible research.
1 INTRODUCTION
LiP-Map addresses unreliable 3D line mapping by jointly optimizing planar surfaces and line structures from posed multi-view images. This interaction produces accurate, complete line maps and improves line-assisted localization.
- Motivation: Existing surface-based lifting often fails because view-dependent parallax creates multiple possible depths and inconsistent depth maps introduce spurious structural lines.The problem is especially difficult because true 3D lines are inseparable from structural surface edges.
- Approach: LiP-Map explicitly associates detected 2D line segments with learned 3D planar edges to recover reliable and consistent 3D line maps.The method is named Line–Plane Joint Optimization for 3D line mapping.
- Approach: The method jointly optimizes planes against depth and normal maps while refining planar structure under supervision from multi-view detected 2D lines.This formulation enforces plane alignment with input geometry and consistency between plane boundaries and observed line segments.
- Results: LiP-Map achieves state-of-the-art accuracy and completeness in 3D line mapping across public indoor-scene benchmarks.Experiments include 50 ScanNetV2 scenes, 30 ScanNet++ scenes, and 10 Hypersim scenes, with additional qualitative evaluation on 7Scenes and Tanks&Temples.
- Results: Integrating LiP-Map primitives into a point-only localization pipeline substantially improves pose accuracy and robustness on 7Scenes.The line correspondences complement point features in textureless or repetitive regions and outperform point-only and existing point–line joint methods.
2 RELATED WORK
The related work situates 3D line mapping within structured reconstruction and motivates LiP-Map’s explicit optimization of the synergy between planes and line segments. The method targets efficient, matching-free reconstruction with structured planar topology.
- 3D Line Mapping: 3D line mapping is useful for indoor scenes where textureless surfaces make traditional keypoint-based mapping suboptimal.Recent advances in line detection and matching have enabled adapting point-mapping pipelines to line features.
- Structured 3D Surface Reconstruction: Structured 3D reconstruction represents scenes with primitives such as planes, surfels, meshes, sketches, and implicit surfaces.PlanarSplatting is highlighted for explicitly optimizing indoor surface geometry from 2.5D cues at ultrafast speed.
- LiP-Map: LiP-Map adopts planar splatting because of its computational efficiency and the intrinsic relationship between plane and line representations.The approach belongs to matching-free solutions.
- LiP-Map: The paper presents explicit optimization of the synergy between 3D line segments and planes as a novel perspective on structured 3D reconstruction.Its pipeline is described as exploring structural synergies between 3D planes and lines.
3 LiP-Map: LINE-PLANE JOINT MAPPING
LiP-Map reconstructs 3D lines as edges of learnable rectangular plane primitives, jointly optimizing planar surfaces and line-edge geometry from multi-view 2D and 2.5D cues. It assigns detected 2D lines to plane edges and constrains those edges in both image and 3D space.
- Problem formulation: LiP-Map takes posed multi-view images with depth, normal, and detected 2D line cues as input for 3D line-map reconstruction.Depth and normal maps may be provided as ground truth or predicted by pretrained 3D foundation models.
- 3.1 Representation of Planar Surface: 3D lines are represented as a subset of the edges of learnable 3D rectangular planes, linking planar surfaces with line structures.Plane interiors approximate scene surfaces, while their edges encompass target 3D lines.
- 3.1 Representation of Planar Surface: The joint formulation simplifies line mapping into optimizing plane edges aligned with 2D detections together with the plane surfaces.The framework explicitly constructs interactions between plane and line primitives rather than treating lines independently.
- 3.2 2D Line to 3D Plane Edge Assignment: For each detected 2D line, LiP-Map associates pixels in a 1-pixel region with the nearest valid intersected planar primitive and then selects a corresponding projected plane edge.Candidate edges are filtered first by angular distance and then by orthogonal endpoint distance.
- 3.3 Line-Plane Joint Optimization: Assigned line-edges are jointly optimized with planes using on-surface, 2D alignment, endpoint, orthogonal, and group constraints.The group loss encourages line-edges assigned to the same 2D segment to remain close in 3D space.
- 3.4 Line Mapping Finalization: After convergence, each 3D line is extracted from an optimized plane edge and retained when its angular and orthogonal distances satisfy the final thresholds.The experiments use τd = 1 pixel and τα = 0.01.
4 EXPERIMENTS
Experiments show that LiP-Map reconstructs accurate, complete, and detailed 3D line maps across indoor benchmarks, while its plane-line design improves robustness, localization, and scalability.
- 3D Line Mapping: LiP-Map achieves a strong balance between 3D line accuracy and completeness, with higher precision, recall, and F1 scores on ScanNetV2 and ScanNet++.It reconstructs more complete line structures, while LIMAP and CLMAP can produce more accurate lines overall.
- Qualitative Comparisons: LiP-Map reconstructs more complete structures than LIMAP and CLMAP under relatively sparse ScanNet++ views.The competing methods reconstruct whole-scene structure on ScanNetV2 with dense views exceeding 100 images but fail to achieve comparable completeness on sparse ScanNet++ views.
- 3D Line Mapping: On Hypersim, LiP-Map obtains lower accuracy and completeness errors and higher precision, recall, and F1 scores than competing methods.It also achieves higher length recall, although LIMAP with depth has slightly higher precision and stronger long-track construction.
- Ablation Studies: Hybrid losses increase the number of reconstructed 3D line segments, reduce completeness error, and improve recall and F1 scores over the PlanarSplatting baseline.The method maintains comparable reconstruction precision while improving 3D line mapping accuracy.
- Ablation Studies: Combining angular consistency with endpoint-distance proximity yields the best 2D-line-to-3D-edge assignment performance.Using either assignment cue individually also improves accuracy, completeness, and the formation of longer line tracks.
- Line-Assisted Visual Localization: Adding LiP-Map’s 3D line map substantially reduces pose estimation error and improves point-based visual localization on 7Scenes.The evaluation reports median translation error, median rotation error, and pose accuracy below 5 cm / 5 degrees.
5 CONCLUSION
LiP-Map jointly optimizes 3D planes and lines using multi-view line, depth, and normal representations. It produces accurate, complete, and detailed line maps while bypassing 2D line matching and supporting multiple line detectors.
- LiP-Map is a line-plane joint optimization framework for 3D line mapping.
- The method optimizes the geometric synergy between 3D planes and line segments using 2D line sketches and 2.5D depth and normal representations.
- Online optimization simultaneously produces 3D line maps and reconstructed planes with prominent topological structure.
- LiP-Map bypasses 2D line segment matching and supports both classical and learning-based 2D line detectors.
- The resulting line maps are accurate, complete, and detailed across real-world and synthetic indoor scene benchmarks.
APPENDIX A MORE IMPLEMENTATION DETAILS
LiP-Map constructs plane intersections, splatting weights, and blended renderings to supervise planar primitives. It also computes scale-invariant geometric consistency measures linking projected planar edges with detected 2D lines.
- Plane Rendering: Ray-to-plane intersection finds where camera rays meet planar primitives before validity checks and rendering.The pipeline uses ray origin and direction to calculate intersections with planes.
- Plane Rendering: Plane splatting assigns intersection weights from projected distances to each primitive’s X- and Y-axes.Sigmoid-based weights are combined using the larger axis-specific value, with λ increased during optimization up to 300.
- Plane Rendering: Planar blending filters intersections below weight 0.0001, sorts them by depth, and selects the five nearest intersections per ray.The selected intersections are used to render depth and normal maps.
- Geometric Consistency: Three scale-invariant measures compare projected planar edges with detected 2D lines using angle distance, orthogonal endpoint distance, and overlap ratio.These measures support geometric constraints for extracting 3D line maps and building line tracks.
- Plane Rendering: Rendered depth and normal maps are supervised using ground-truth or Metric3Dv2 and Omnidata pseudo-labels.The corresponding render loss uses α1 = 5.0, α2 = 1.0, and α3 = 2.0.
APPENDIX C COMPUTATION OF EVALUATION METRICS
The appendix defines M1 and M2 by sampling reconstructed 3D lines and comparing them with ground-truth meshes or point clouds through nearest-neighbor distances. M1 evaluates point-level accuracy and completeness, while M2 evaluates the proportion and length of tracks near the ground truth.
- Metric M1: M1 computes accuracy and completeness from bidirectional nearest-neighbor distances between reconstructed samples and the ground-truth model.Precision, recall, and F1 use a distance threshold τd of 0.05m.
- Metric M1: Metric M1 samples 100 points per line for line-level evaluation and uses line endpoints for junction-level evaluation.KD-trees over predicted samples and the ground-truth point cloud provide closest distances.
- Metric M2: Metric M2 samples 1000 points along each reconstructed 3D line and measures the proportion of each line within selected distance thresholds from the ground-truth model.The reported thresholds are 1/5/10 mm or 5/10/50 mm, depending on the evaluation.
- Metric M2: M2 converts qualifying sampled portions into total line length within τd of the ground-truth model.This length-based evaluation complements the per-track proximity proportion.
APPENDIX D GLOBAL LINE MERGING
Global line merging addresses detector-induced fragmentation and duplication by associating 2D detections with plane-derived 3D lines, clustering them, propagating identifiers, and merging each group. Qualitative results report improved line-map quality and completeness, while planar reconstruction also outperforms PlanarSplatting.
- Results: Compared with no merging, local merging removes incorrect lines, while global merging further consolidates corresponding lines and improves line-map quality and completeness.Global merging is more computationally expensive than local merging.
- Global Merging: The global merging pipeline casts rays from one-pixel detection regions and collects edges from intersected 3D planes.It then clusters associated lines using pairwise perpendicular distances sampled at 10 points per line and DBSCAN.
- Global Merging: Global identifier propagation assigns existing or new identifiers to each DBSCAN cluster before grouping 3D lines.Each identifier defines a candidate merged-line group.
- Global Merging: PCA-based merging estimates a group direction and mean point, projects endpoints onto that line, and constructs merged endpoints from the minimum and maximum projections.The result is added to the merged 3D line map.
- Planar Reconstruction: LiP-Map achieves superior planar reconstruction results to PlanarSplatting on ScanNetV2 and aligns planar primitive edges with scene line structures.The resulting meshes show more prominent geometric topology and support complete, precise 3D line extraction.
APPENDIX F LIMITATIONS
The method’s limitations arise from duplicated segments caused by 2D detection errors and from the plane-based representation’s reduced suitability for non-planar, unstructured objects. The authors position the approach primarily for architectural and man-made environments.
- Detection Errors: 2D line-detection errors can create duplicate 3D line segments that violate parsimony in the reconstructed line map.The limitation is linked to fragmented, duplicated, or misaligned detections across views.
- Scope: The plane-based representation is particularly suited to architectural structures but has limited capability for non-planar, unstructured objects.The stated scope boundary concerns the representation’s fit to scene geometry.
APPENDIX G MORE QUALITATIVE RESULTS OF OUR LINE MAPS
The appendix presents additional qualitative 3D line-map results across ScanNetV2, ScanNet++, and Hypersim scenes.
- Additional qualitative 3D line-map results are shown for ScanNetV2 scenes.
- Additional qualitative 3D line-map results are shown for ScanNet++ scenes.
- Additional qualitative 3D line-map results are shown for Hypersim scenes.
APPENDIX H QUALITATIVE RESULTS ON THE SENSITIVITY TO HEURISTICS AND HYPERPARAMETERS
The appendix examines sensitivity to plane-splitting, initialization, assignment, and loss-weight settings, while testing whether the joint optimization contributes beyond input priors.
- Heuristics and hyperparameters: The plane-splitting threshold balances excessive plane fragmentation against insufficient fine-grained reconstruction.A small threshold can cause the number of planes to explode, whereas a large threshold can leave too few planes.
- Heuristics and hyperparameters: Larger initial plane counts are necessary for better reconstruction on large-scale scenes.The appendix reports corresponding optimization-time results and qualitative comparisons.
- Heuristics and hyperparameters: Strict assignment thresholds, τd = 1 pixel and τα = 0.01 rad, prioritize accurate 3D line mapping over completeness.Relaxing them produces more complete maps but also many noisy lines.
- Heuristics and hyperparameters: The chosen balancing weights αΠ = 10 and αL = 0.1 place the planar and line losses on comparable numerical scales.A large αL can rapidly increase the number of planes and optimization time.
- Input-prior robustness: Experiments with varied depth and normal-map quality indicate robustness to some input noise and sparsity beyond merely fitting priors.The appendix evaluates ScanNetV2 and Hypersim inputs from multiple depth and normal-map sources.
APPENDIX J MORE RESULTS ABOUT GENERALIZATION TO NON-PLANAR STRUCTURES
The method is evaluated on non-planar Hypersim scenes, where it approximates curved structures but can produce inaccurate or hallucinated planes locally.
- Generalization to non-planar structures: The method approximates curved Hypersim structures, including chairs and circular tables, using reconstructed meshes and 3D line maps.
- Generalization to non-planar structures: Local non-planar failures include inaccurate or hallucinated planes caused by noisy depth or normal-map supervision.
- Generalization to non-planar structures: Strict assignment thresholds filter some inaccurate plane edges by enforcing geometric consistency between 2D and 3D lines.
APPENDIX K FAILURE CASES
The appendix catalogs complete and partial failures under challenging camera-motion, initialization, non-planar, and noisy-input conditions, with accompanying qualitative visualizations.
- Failure conditions: Purely rotational camera sequences can cause reconstruction failure because unreliable outputs produce severe geometric drift in the initial mesh.The appendix describes this as a classic ill-posed problem.
- Failure conditions: Insufficient initial plane primitives can cause reconstruction failure for large-scale scenes.
- Failure conditions: Non-planar scenes remain challenging, while noisy or inaccurate initial meshes or depth maps can degrade reconstruction quality.
- Qualitative visualizations: The appendix visualizes additional qualitative results and failure cases across multiple datasets and reconstruction settings.Figures cover ScanNetV2, ScanNet++, Hypersim, threshold variations, loss-weight variations, mesh quality, and non-planar structures.