PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable and Scalable Bundle Adjustment of Holistic 3D Structures

Shaohui Liu, Rémi Pautrat, Daniel Barath, Richard Hartley, Viktor Larsson, Marc Pollefeys

arXiv:2609.04026v1cs.CV

TL;DR

Richer geometric structures make bundle adjustment harder by coupling features and increasing computational cost and instability. The paper introduces a unified reprojection-based framework for features, higher-order groups, and cross-feature relations, achieving comparable runtime with richer reconstructions and improved accuracy.

  • Problem

    Integrating structures such as parallelism, coplanarity, and wireframes into bundle adjustment can increase computational cost and reduce numerical stability because geometric relations couple landmark parameters.

  • Method

    The framework classifies entities as directly measured features or higher-order groups, models groups alongside cameras, and expresses group and cross-feature constraints as 2D reprojection residuals.

  • Results

    The method achieves runtime comparable to classical bundle adjustment while producing significantly richer reconstructions and improving camera poses and geometry across synthetic and real-world evaluations.

  • Takeaways & Limitations

    Reprojection-based group and cross-feature constraints preserve efficient Schur elimination while avoiding the scale and conditioning problems of direct 3D regularization.

  • Takeaways & Limitations

    The formulation cannot recover features with degenerate viewing geometry onto the group surface, although such features have little influence on group estimation.

Abstract

from arXiv · show

Bundle Adjustment (BA) is a cornerstone of 3D computer vision and has benefited from decades of advances in sparse optimization and numerical methods. It was originally developed for jointly optimizing camera intrinsics, poses and sparse 3D points. While extensions incorporate lines and other primitives, integrating richer geometric structures such as parallelism, coplanarity, or wireframes often introduces significantly increased computational cost and reduced numerical stability. In this paper, we propose a unified framework that extends bundle adjustment to jointly optimize geometric features and higher-order relations. We first introduce a taxonomy that distinguishes scalable geometric features with direct 2D measurements (e.g., points and lines), from groups encoding higher-order relations (e.g., coplanarity, parallelism, etc.), where we show that groups can be modeled as camera-like entities within the bundle adjustment framework. Building on this formulation, we propose that both group constraints and cross-feature relations (i.e., point-line associations) can be expressed through 2D reprojection measurements. By formulating group-induced and cross-feature reprojection errors, we preserve the sparsity structure of classical point-based BA under Schur elimination, while avoiding direct 3D regularization that degrades the conditioning and stability. Experiments on both real-world and synthetic datasets demonstrate runtime performance comparable to classical point-only bundle adjustment, while producing significantly richer 3D structures and improved geometric accuracy.

1 Introduction

The paper extends bundle adjustment to richer geometric structures while addressing the sparsity and conditioning problems introduced by structural relations. Its unified reprojection-based formulation preserves efficient optimization and improves reconstruction richness and accuracy at comparable runtime.

  • 1 Introduction: Structural relations such as coplanarity, parallelism, and wireframe connectivity couple landmarks, creating denser Hessians and potentially undermining efficient Schur elimination.Direct 3D penalties also mix metric and pixel-space residuals, causing scale mismatch, ad hoc weighting, and reconstruction shrinkage.
  • 1 Introduction: The framework distinguishes directly measured features, such as points and lines, from higher-order groups, such as vanishing points, planes, and conics.Groups are placed alongside cameras in the non-eliminated normal-equation block, absorbing feature coupling while preserving block-diagonal structure over features.
  • 1 Introduction: Group and wireframe constraints become pixel-space reprojection residuals, avoiding direct 3D regularization and weighting constraints through feature uncertainty.Group-induced residuals measure projected displacement onto a group surface, while cross-feature residuals use one feature's observation to constrain another.
  • 1 Introduction: Experiments on synthetic and real-world datasets achieve runtime comparable to classical bundle adjustment while optimizing richer holistic 3D reconstructions and improving camera poses and geometry.The framework extends sparse reconstructions beyond point clouds to include geometric primitives such as planes, spheres, and cylinders.

2 Related Work

Prior work has advanced sparse bundle adjustment and optimized individual geometric primitives, lines, planes, and wireframes. However, existing formulations often do not integrate structural relationships and supporting features seamlessly with classical point-based bundle adjustment.

  • 2 Related Work: Sparse matrix techniques and Schur complements exploit camera–point bipartite structure, while later work improves bundle-adjustment scalability through parallel solvers, robust costs, and incremental formulations.These advances target efficient optimization of large-scale reconstruction problems.
  • 2 Related Work: Geometric-primitive methods optimize high-dimensional primitive variables from 2D observations but typically ignore structural relationships between primitives and their supporting features.This limits seamless integration with classical bundle adjustment over 3D points.
  • 2 Related Work: Line-based optimization uses orthonormal Plücker coordinates and incorporates cues including vanishing points, Manhattan-world assumptions, and learned monocular priors.These systems build on extensive prior work treating 3D lines as optimization primitives.
  • 2 Related Work: Plane optimization includes homography, auto-calibration, and explicit 3D-structure formulations, with recent work revisiting homography constraints for computational efficiency.Homography-based approaches require enumerating image pairs, while explicit parameterizations directly represent 3D structures.
  • 2 Related Work: Deep networks enable 2D wireframe prediction, and wireframes support feature matching through junction-aware and learned point–line association methods.Recent line-based reconstruction systems also incorporate point–line associations using 3D regularization.

3 A Unified Framework of Holistic Bundle Adjustment

The framework extends bundle adjustment to richer geometric structures by separating directly measured features from higher-order groups and expressing their constraints through reprojection errors. This preserves feature-block sparsity for Schur elimination while integrating group and cross-feature relations.

  • 3.1 Review of Classical Bundle Adjustment: Classical bundle adjustment minimizes reprojection errors and exploits block-diagonal point Hessians to eliminate structure efficiently into a camera-only system.Each feature contributes independently to the feature block, enabling efficient Schur complement elimination.
  • 3.2 Features and Groups: A Taxonomy on Sparse 3D Structures: Groups are modeled as camera-like variables in the non-eliminated block, so feature–group residuals preserve the block-diagonal structure of the feature Hessian.This absorbs coupling through the group variables while retaining efficient Schur elimination over features.
  • 3.2 Features and Groups: A Taxonomy on Sparse 3D Structures: The taxonomy distinguishes features with direct 2D measurements, such as points and lines, from groups encoding higher-order geometric relations.The formulation treats these entity types differently according to their roles in the normal equations.
  • 3.3 Group Constraints in Bundle Adjustment: Positional group constraints use pixel-space differences between ordinary and group-projected reprojections instead of direct 3D penalties.The general construction supports entities with projection operators onto constraint surfaces and retains one-feature, one-group, and one-camera residual structure.
  • 3.4 Cross-Feature Constraints in Bundle Adjustment: Cross-feature reprojection errors enforce wireframe relations using one feature as an optimization variable and the other feature’s 2D observation as a fixed measurement.Each residual involves only one feature and one camera, preserving block-diagonal feature structure while extending the effective visibility graph.
  • 3.5 Discussions and Applications: The framework integrates features, groups, and wireframe edges into incremental structure-from-motion while preserving efficient elimination and improving pose and geometry accuracy.The authors report runtime comparable to classical bundle adjustment and consistent accuracy improvements, while noting that degenerate viewing geometry cannot be recovered onto a group surface.

4 Experiments

Experiments evaluate runtime scaling, reconstruction quality, and full SfM performance for increasingly rich geometric bundle-adjustment configurations. The framework preserves classical BA scaling while improving geometry and producing richer holistic maps.

  • Benchmarking Holistic Bundle Adjustment: All configurations scale similarly, approximately n^1.6−1.9 with SPARSE_SCHUR and n^2.6−2.8 with DENSE_SCHUR, preserving classical BA complexity.The configurations are Point, Point-Line, Groups, and Holistic.
  • Benchmarking Holistic Bundle Adjustment: 2D reprojection-based wireframe constraints are moderately faster than 3D endpoint-distance constraints with SPARSE_SCHUR, with a wider gap under DENSE_SCHUR.The 3D formulation creates denser fill-in in the reduced system.
  • Applications: In full SfM experiments, the framework achieves comparable runtime and reprojection error while registering comparable or slightly more images and improving camera poses and geometry.The framework is evaluated on real-world and synthetic data, with pose accuracy reported using relative pose AUCs.
  • Benchmarking Holistic Bundle Adjustment: Group constraints improve reconstructed point accuracy and substantially increase line recall and precision, while wireframe constraints provide further gains.The evaluation uses eight Hypersim scenes and compares points and lines against ground-truth meshes.
  • Benchmarking Holistic Bundle Adjustment: On ETH3D, the method improves point reconstruction accuracy and completeness across all evaluation thresholds.The evaluation follows the official multi-view benchmark against laser scans.
  • Applications: The ablation finds that 2D group-induced reprojection requires no weight tuning and achieves the best or competitive geometry and pose results.Direct 3D plane-distance constraints require careful tuning; large weights can degrade pose accuracy on ScanNet++.

5 Conclusion

The framework integrates higher-order geometric structures into bundle adjustment while preserving efficiency. It achieves comparable runtime, richer reconstructions, and improved accuracy.

  • The proposed framework integrates higher-order geometric structures while preserving efficient bundle adjustment.
  • Experiments show comparable runtime with significantly richer reconstructions and improved accuracy.
  • Group parameters and constraints are handled through a unified bundle adjustment formulation.

Appendix

The supplementary material documents implementation details, group parameterizations, framework discussions, and additional visualizations.

  • The supplement begins with implementation details for image description, matching, structure-from-motion, and the synthetic benchmark.
  • It presents example groups and their parameterizations in Section B.
  • It concludes with additional visualizations after discussing the proposed bundle adjustment framework.

A.1 Additional Details on Image Description and Matching

The image-description pipeline combines specialized detectors and matchers for points and lines with methods for vanishing points, planes, and wireframe junctions.

  • ALIKED detects points and LightGlue matches them, while DeepLSD detects lines and GlueStick matches them.
  • JLinkage detects vanishing points, and planes are segmented by fitting predicted depth and surface normals from MoGe-2.
  • Wireframe junctions are derived from spatial incidences between detected points and lines within a 2-pixel distance threshold.

A.2 Additional Details on Structure-from-Motion

The supplementary structure-from-motion pipeline uses image retrieval, incremental triangulation, group-support filtering, and COLMAP-based local and global bundle adjustment.

  • A.2 Additional Details on Structure-from-Motion: Top 30 nearest neighbors are retrieved for each image using NetVLAD for image pair selection.
  • A.2 Additional Details on Structure-from-Motion: Figure 5 provides examples of detected features, groups, and wireframes.
  • A.2 Additional Details on Structure-from-Motion: Points and lines are triangulated first, followed by incremental group triangulation as new images are registered.
  • A.2 Additional Details on Structure-from-Motion: The pipeline extends COLMAP principles with next-image selection, registration, triangulation, local bundle adjustment, and periodic global bundle adjustment.

A.3 Details on Setup of the Synthetic Benchmark

The synthetic benchmark isolates scaling behavior using cameras surrounding a cubic scene containing Manhattan-aligned planes, points, lines, and wireframe junctions.

  • A.3 Details on Setup of the Synthetic Benchmark: Cameras surround a cubic scene with inward viewing directions to evaluate bundle-adjustment scaling in isolation.The cube has halfextent 3.0.
  • A.3 Details on Setup of the Synthetic Benchmark: The benchmark includes randomly offset planes whose normals align with the x, y, or z axes.These planes represent Manhattan-aligned structure.
  • A.3 Details on Setup of the Synthetic Benchmark: Points and lines are distributed both on planes and as free-floating features, while wireframe junctions connect regular-grid points sampled on each plane.

B Example Groups and Their Parameterizations

The framework supports geometric groups that constrain points and lines to surfaces or directions, using explicit parameterizations and projection operations for diverse primitives.

  • B Example Groups and Their Parameterizations: Supported groups constrain 3D points and lines to surfaces or directional structures, with positive quantities optimized in log-space.Table 6 summarizes each group’s illustration, degrees of freedom, and constrained features.
  • B Example Groups and Their Parameterizations: Planes use a unit normal and signed offset, projecting points by removing their signed normal distance from the plane.
  • B Example Groups and Their Parameterizations: For line constraints, projected directions remove the component along the plane normal, and projected moments use the associated Plücker-coordinate expression.
  • B Example Groups and Their Parameterizations: Additional visualizations show bundle-adjusted holistic structures with parallel lines sharing colors, while displayed line and plane extents are illustrative only.The optimization treats lines and planes as infinite entities.
  • B Example Groups and Their Parameterizations: Spheres, cylinders, cones, ellipsoids, and cuboids are represented with geometric parameters and corresponding surface or radial projection procedures.Sphere projection uses a center and positive radius; other primitives use their stated axes, poses, offsets, or angles.

C More Details and Discussions

The framework addresses scale sensitivity, connects its constraints to Manhattan and Atlanta world assumptions, and finds that wireframe constraints modestly densify the Schur complement.

  • C More Details and Discussions: Angular vanishing-point costs are scale-free, unlike 3D group and wireframe costs whose magnitudes depend on global scene scale and viewing geometry.
  • C More Details and Discussions: Vanishing-point line constraints and inter-group orthogonality constraints implicitly enforce Manhattan or Atlanta world structure.The Manhattan assumption uses three orthogonal dominant directions, whereas Atlanta world allows multiple horizontal directions around a vertical direction.
  • C More Details and Discussions: Wireframe constraints increase Schur-complement density by only 2–7% relative to baseline across three 1DSfM scenes.New camera correlations arise only for particular non-co-observed feature configurations, making the practical density increase modest.
  • C More Details and Discussions: Additional visualizations present bundle-adjusted holistic 3D structures from Hypersim and ScanNet++.Parallel lines are colored consistently in the visualizations.
Loading 2609.04026v1…