PaperlayerView exact revision on arXiv

Source-linked AI summary

PLANING: A Loosely Coupled Triangle-Gaussian Framework for Streaming 3D Reconstruction

Changjian Jiang, Kerui Ren, Xudong Li, Kaiwen Song, Guanghao Li, Linning Xu, Tao Lu, Junting Dong, Yu Zhang, Bo Dai, Mulin Yu

arXiv:2601.22046v4cs.CV

TL;DR

Streaming monocular 3D reconstruction must reconcile high-quality rendering with accurate, efficient geometry. PLANING addresses this with a loosely coupled triangle–Gaussian representation and streaming-aware optimization, reporting state-of-the-art reconstruction quality, speed, and structural compactness across benchmarks. Its limitations include transparency handling and outdoor sky or distant-background modeling.

  • Problem

    Streaming monocular reconstruction methods rarely achieve both high-quality rendering and accurate geometry, while Gaussian-based approaches can lack compact, stable structural geometry.

  • Method

    PLANING loosely couples explicit learnable triangles for geometry with neural Gaussians for appearance, using streaming-aware initialization, global map adjustment, and gradient-based geometric refinement.

  • Results

    PLANING demonstrates state-of-the-art geometric accuracy and rendering quality across diverse indoor and outdoor benchmarks while reducing training time and primitive count.

  • Takeaways & Limitations

    The compact, structurally clear output supports large-scale reconstruction, pose refinement, and simulation-ready environments for embodied AI.

  • Takeaways & Limitations

    Neural Gaussians are poorly suited to semi-transparent or transparent objects, and outdoor sky or distant-background regions can cause inconsistent initialization and degraded appearance quality.

Abstract

from arXiv · show

Streaming reconstruction from monocular image sequences remains challenging, as existing methods typically favor either high-quality rendering or accurate geometry, but rarely both. We present PLANING, an efficient on-the-fly reconstruction framework built on a hybrid representation that loosely couples explicit geometric primitives with neural Gaussians, enabling geometry and appearance to be modeled in a decoupled manner. This decoupling supports an online initialization and optimization strategy that separates geometry and appearance updates, yielding stable streaming reconstruction with substantially reduced structural redundancy. PLANING improves dense mesh Chamfer-L2 by 18.52% over PGSR, surpasses ARTDECO by 1.31 dB PSNR, and reconstructs ScanNetV2 scenes in under 100 seconds, over 5x faster than 2D Gaussian Splatting, while matching the quality of offline per-scene optimization. Beyond reconstruction quality, the structural clarity and computational efficiency of PLANING make it well suited for a broad range of downstream applications, such as enabling large-scale scene modeling and simulation-ready environments for embodied AI. Project page: https://city-super.github.io/PLANING/ .

1. Introduction

PLANING targets streaming 3D reconstruction’s need for both geometric accuracy and real-time efficiency with a hybrid representation that separates geometry from appearance. Its experiments report broad gains in reconstruction quality, efficiency, and compactness while supporting structured planar geometry.

  • Motivation: 3D scene reconstruction supports spatial understanding for embodied intelligence, autonomous driving, and AR/VR, but offline post-processing limits scalability and responsiveness.The introduction motivates real-time, on-the-fly reconstruction for time-critical scenarios.
  • Motivation: Existing streaming Gaussian methods provide effective appearance modeling but lack compact, stable geometry, causing redundancy and weaker geometric consistency.Their appearance-focused optimization and large primitive counts limit scalability, especially under sparse observations or novel viewpoints.
  • Method: PLANING loosely couples learnable triangle primitives for geometry with neural Gaussians for appearance.Triangles provide explicit surface structure, while neural Gaussians support high-fidelity rendering through fused triangle and per-Gaussian features.
  • Results: PLANING outperforms existing methods across geometry accuracy, rendering quality, computational efficiency, and memory usage while preserving clear planar geometry.Figure 2 summarizes the cross-axis comparison, and the reported experiments span indoor and outdoor benchmarks.
  • Method: The framework combines the hybrid representation with streaming-aware initialization and global map adjustment for efficient on-the-fly monocular reconstruction.The design uses photometric and spatial filtering to reduce redundant primitives and maintain global consistency with optimized camera poses.

2. Related Work

Prior reconstruction research spans implicit neural rendering, explicit Gaussian representations, geometry-aware primitives, and streaming SLAM systems. PLANING is positioned against limitations involving computational cost, geometric control, and the balance among robustness, fidelity, and efficiency.

  • 3D Reconstruction: Traditional reconstruction and NeRF established strong geometric or rendering foundations, but implicit fields, mesh extraction, and per-ray sampling hinder scalability or geometric control.NeRF-based methods are limited by costly per-ray volumetric rendering and implicit representations.
  • 3DGS Variants: 3DGS variants replace or augment Gaussian primitives with ellipsoids, quadrics, convexes, triangles, or rectangles to better align representations with scene geometry.These alternatives target surface alignment or compact modeling of hard-edged scenes.
  • 3DGS Variants: Some geometry-aware Gaussian designs primarily enhance appearance rather than explicitly decoupling geometry and appearance.This distinction motivates PLANING’s loosely coupled triangle–Gaussian representation.
  • Streaming Reconstruction: Classical visual SLAM offers robust online tracking and mapping but often lacks the fidelity needed for high-quality rendering.NeRF-based SLAM improves photorealism but remains computationally costly for real-time applications.
  • Streaming Reconstruction: 3DGS-based SLAM improves rendering efficiency, yet monocular systems still struggle to balance robustness, reconstruction accuracy, and efficiency.Recent approaches combine GPU-friendly mini-bundle adjustment with incremental 3DGS updates.

3. Method

PLANING combines learnable triangles for explicit geometry with neural Gaussians for appearance in a streaming monocular reconstruction framework. Its rasterization, pose-processing, primitive-initialization, and map-update components support efficient, compact reconstruction.

  • Dual Scene Representation: PLANING combines learnable triangle primitives for geometry with neural Gaussians for appearance in a dual scene representation.Triangles provide explicit surface structure, while Gaussians are anchored to triangles and decoded using associated features.
  • Learnable Triangles for Geometry: Each triangle is parameterized by three learnable vertices and a local coordinate frame, with additional parameters controlling edge sharpness, boundary smoothness, and opacity.The local frame uses the triangle barycenter as origin and represents vertices in the local tangent plane.
  • Differentiable Triangle Rasterizer: The differentiable triangle rasterizer computes triangle intersections and front-to-back compositing for depth, normals, and appearance rendering.It uses explicit ray-triangle intersections, edge-preserving contributions, visibility determination, and triangle-subdivision-based depth sorting.
  • Neural Gaussians for Appearance Modeling: Neural Gaussians are positioned relative to triangle barycenters, with scales and rotations predicted from concatenated triangle and Gaussian features.Each triangle can host a flexible number of Gaussians, allowing local adaptation to scene details.
  • Streaming Reconstruction Framework: The streaming framework processes unposed monocular sequences through camera tracking, global pose optimization, and scene mapping.The backend performs loop closure and global bundle adjustment, while global map adjustment follows camera-pose updates.
  • Primitive Initialization and Map Compactness: Primitive insertion is restricted to poorly covered or highly erroneous regions, while depth-adaptive spatial filtering prevents redundant growth near existing triangles.Photometric filtering uses image discrepancies and a threshold; spatial filtering removes candidates within a depth-dependent vicinity.

4. Experiments

PLANING is evaluated across planar, dense-mesh, rendering, large-scale, locomotion, and ablation settings. Results show strong geometric and rendering quality with reduced redundancy, efficient training, and useful downstream scene properties.

  • Experimental Setup: PLANING is evaluated on 56 real-world indoor and outdoor scenes spanning ScanNet++, ScanNetV2, VR-NeRF, FAST-LIVO2, KITTI, and Waymo.Comparisons cover per-scene, streaming, and planar reconstruction baselines, with task-specific geometric and rendering metrics.
  • Geometry Results: PLANING achieves top-tier planar reconstruction while significantly reducing primitive count and runtime across ScanNet++, ScanNetV2, and FAST-LIVO2.The evaluation reports the lowest Chamfer Distance and highest F-score while maintaining compact representation and short training time.
  • Geometry Results: PLANING achieves higher dense-mesh geometric accuracy while requiring less than 20% of the training time of per-scene optimization methods.All dense meshes are extracted via depth fusion for fair comparison.
  • Rendering Results: PLANING achieves state-of-the-art rendering performance across diverse datasets, with advantages in texture-less and low-light regions.The method uses a precise geometric model and feed-forward pose estimation to mitigate initialization and pose-drift artifacts.
  • Framework Design: Plane-guided pose optimization feeds reconstructed planar maps into the frontend, using online plane extraction and point-to-plane alignment to improve global consistency.Planar primitives provide stable geometric supervision for pose estimation.
  • Large Scale Scene Reconstruction: Dynamic loading swaps primitive parameters between GPU and CPU, enabling high-quality reconstruction of large indoor environments under limited GPU memory.A corridor sequence with over 2000 monocular images demonstrates the large-scale setting.
  • Efficient Locomotion Strategy Training: The reconstructed planar scenes provide reliable contact geometry for lightweight simulation and scalable locomotion-policy training.Experiments include indoor walking with a Unitree H1 and stair climbing with a Unitree A1 using PPO in Isaac Lab.
  • Framework Design: +200% primitives on ScanNetV2 and +245% on ScanNet++ result when spatial filtering is disabled, confirming its role in reducing redundancy.Disabling the global map update improves geometric consistency and rendering quality.

5. Limitations

PLANING inherits limitations from its primitives and scene assumptions. Neural Gaussians are unsuitable for some transparent content, while outdoor sky and distant backgrounds are not explicitly handled.

  • Scope Boundary: Neural Gaussian primitives are poorly suited to semi-transparent or transparent objects, where unreliable appearance gradients may adversely affect geometry optimization.This limitation follows from the chosen primitives and scene assumptions.
  • Scope Boundary: PLANING focuses on surface modeling and does not explicitly handle sky or distant background regions in outdoor scenes.These regions can cause inconsistent initialization and degraded appearance quality.

6. Conclusion

PLANING combines loosely coupled triangle geometry with neural Gaussians and subdivision-aware rendering to support stable, structurally robust streaming reconstruction.

  • PLANING decouples geometry from appearance using a loosely coupled triangle–Gaussian representation that preserves high-fidelity rendering.The framework uses triangles as anchoring geometry while neural Gaussians model appearance.
  • Triangle primitives provide robust geometric anchors that address drift, redundancy, and instability in streaming Gaussian-based reconstruction.The representation targets the absence of compact, stable anchoring geometry in existing frameworks.
  • Large triangles can produce inaccurate depth sorting and visibility estimates when barycenters poorly represent ray intersections or occlusion.These issues arise when triangle barycenters deviate from true intersections or are occluded while portions remain visible.
  • Subdivision-aware forward rendering uses adaptive triangle subdivision for depth sorting and vertex-based visibility for occlusion handling.The complete rendering procedure is summarized in Algorithm 1.

A.2. Training Strategy

The streaming system uses staged optimization and a subdivision-aware forward-rendering algorithm that preprocesses visible triangles, subdivides them, sorts them, and renders depth and normals.

  • Keyframes initialize new primitives and run M=20 optimization iterations, while common frames run M/2 iterations without adding Gaussians.Training samples draw with probability 0.2 from the current frame and 0.8 from past frames.
  • A coarse-to-fine plane-extraction strategy progressively detects smaller planes from residual points after coarser planes are extracted.The procedure is applied iteratively with progressively finer parameters.
  • Algorithm 1 takes a triangle soup, camera pose, and screen resolution as input and outputs rendered depth and normal maps.The pipeline begins by initializing a visible-triangle set and processing triangles with visible vertices.
  • Visible triangles are transformed into local frames and recursively subdivided until every edge is shorter than threshold ϵ.Subdivision triangles retain their parent triangle IDs before entering subsequent processing.
  • The renderer projects subdivision triangles, computes depth-based sorting keys, performs GPU radix sorting, and evaluates ray–triangle intersections per pixel.It then renders depth and normal images following the specified rendering equations.

A.4. More Implementation Details

Implementation details cover rendering and extraction settings, planar-map pose refinement, dynamic primitive loading, and scene import evaluation.

  • Rendering quality improves initially and then saturates as the number of Gaussians increases.Figure 13 reports this trend for both PSNR and SSIM.
  • The system refines camera poses with point-to-plane alignment using a voxelized planar map and extracts planes every 10 frames.The implementation uses a 3 cm voxel size and region-growing plane extraction.
  • Table 5 compares scene import and conversion time in GUI-based and headless Isaac Sim pipelines performing identical processing steps.Both settings include mesh import, collision geometry construction, and USD packaging.
  • Dynamic loading swaps primitive parameters between GPU and CPU, marking triangles invisible when all associated neural Gaussians are invisible.A Gaussian is invisible when its projected scale is smaller than a pixel.

B.3. Locomotion Strategy Training

PLANING produces compact planar geometry intended for embodied simulation, reducing import overhead and supporting stable locomotion when policies rely on geometric correctness.

  • PLANING prioritizes large-scale load-bearing structures such as floors and walls, producing compact geometry while preserving structural integrity.This differs from appearance-driven methods that generate redundant primitives.
  • A 2DGS scene with 277k faces requires over 30 minutes for standard Isaac Sim import and conversion.PLANING’s lower mesh complexity avoids these preprocessing bottlenecks in standard and headless pipelines.
  • Locomotion experiments evaluate reconstructed geometry with Unitree H1 and A1 policies trained without a height scanner.This setting requires policies to rely on the physical correctness of simulated geometry.
  • PLANING preserves planar structures while capturing fine geometric details across geometric comparisons.Figure 14 compares planar reconstruction and geometric modeling across different primitives, including dense-mesh 2DGS.
  • Policies trained in aggressively simplified 2DGS scenes fail to converge, whereas policies trained in PLANING scenes consistently achieve stable locomotion.The comparison uses identical observation settings and links performance to the reconstructed planar geometry.

C.1. Supplementary Comparison Experiments

Supplementary comparisons evaluate PLANING's reconstruction and rendering quality across ScanNet++/ScanNetV2, KITTI, and VR-NeRF, with results indicating stronger geometric preservation and sharper rendering than baseline methods.

  • The supplementary table organizes ScanNet++ ablations into representation-design and framework-design categories.
  • Additional reconstruction comparisons on ScanNet++ and ScanNetV2 show that PLANING more faithfully preserves scene geometric structures.
  • PLANING demonstrates robustness across indoor and outdoor scenarios in rendering comparisons on KITTI and VR-NeRF.
  • PLANING produces sharper rendering on intricate structures such as text while achieving superior overall rendering quality against baseline methods.

C.2. Supplementary Ablation Studies

Ablation studies on ScanNet++ examine both representation and framework design, showing that spatial filtering improves geometric efficiency while Gaussian scaling eventually saturates rendering gains.

  • Representation design: The representation design achieves the best performance in both rendering quality and geometric accuracy on ScanNet++.
  • Framework design: Spatial filtering delivers higher geometric accuracy and comparable rendering quality with less than one-third of the primitives.
  • Gaussian count: PSNR and SSIM initially improve as the Gaussian count increases and then saturate.
Loading 2601.22046v4…