PaperlayerView exact revision on arXiv

Source-linked AI summary

PIVOT: A Multi-Trajectory Dataset and Testbed for Pose, Intrinsics, and Novel Viewpoint Evaluation in Real-World 3D Reconstruction

Mary Raymond

arXiv:2608.25401v1cs.CVcs.AI

TL;DR

Existing reconstruction benchmarks often assume favorable trajectories, optimized cameras, and views drawn from represented paths, leaving deployment-oriented conditions less directly measured. PIVOT introduces a multi-trajectory dataset and evaluation framework that separates trajectory, pose, and intrinsic sources; its experiments find consistent degradation on unseen trajectories and substantial sensitivity to pose and intrinsics.

  • Problem

    Common evaluations use reconstruction-friendly trajectories, optimized camera parameters, and held-out views from represented trajectories rather than measured poses, reusable calibration, and structurally different paths.

  • Method

    PIVOT captures diverse trajectories while retaining measured and optimized poses, calibrated and optimized intrinsics, three benchmark families, and a directed pose-space Chamfer distance.

  • Results

    Across scenes, unseen trajectories consistently underperform seen held-out views, measured pose components substantially degrade quality, and intrinsic calibration source measurably affects results.

  • Takeaways & Limitations

    Trajectory identity, pose source, and intrinsic source should be treated as explicit evaluation dimensions when studying reconstruction under real-world capture conditions.

Abstract

from arXiv · show

Neural radiance fields (NeRFs), 3D Gaussian Splatting (3DGS), and related novel-view synthesis methods are commonly evaluated under capture and reconstruction conditions cleaner than those encountered by robots, drones, and autonomous systems. Benchmarks often rely on reconstruction-friendly trajectories, optimized camera poses and intrinsics, and held-out views sampled from trajectories represented during training. These assumptions can obscure performance with measured poses, reusable camera calibration, and structurally different camera paths. We introduce PIVOT (Pose, Intrinsics and Viewpoint Oriented Testbed), a multi-trajectory dataset, processing pipeline, and evaluation framework for independently studying these factors. PIVOT captures each scene using diverse camera trajectories and retains, where available, both sensor-derived measured poses and COLMAP-optimized poses, together with calibrated and optimized camera intrinsics. It defines three benchmark families: (1) seen versus unseen trajectory novel-view generalization, (2) measured versus optimized pose sensitivity, and (3) calibrated versus optimized intrinsics sensitivity. We also introduce a directed pose-space Chamfer distance to quantify how well training poses cover an evaluation trajectory. PIVOT v1 contains five real-world scenes captured with a DJI Mini 4 Pro and provides an open processing and Nerfstudio-based evaluation toolchain. Benchmark results show a consistent quality gap between held-out views on represented trajectories and unseen trajectories, as well as substantial sensitivity to pose source and camera intrinsics.

I. INTRODUCTION

PIVOT addresses evaluation settings that are cleaner than real-world capture by treating camera trajectories, pose sources, and intrinsics as explicit experimental factors. It provides a multi-trajectory testbed and controlled benchmarks for studying trajectory generalization, pose sensitivity, and intrinsic sensitivity.

  • Common pipelines use smooth, high-overlap trajectories, offline-optimized poses and intrinsics, and held-out views from represented trajectory families.
  • Robotic and aerial systems may instead use online localization, reusable calibration, traversal motion, and structurally different camera paths.
  • PIVOT preserves trajectory identity together with measured and COLMAP-optimized camera parameters for reproducible controlled evaluation.
  • The testbed investigates generalization to structurally different trajectories, pose-distribution distance, measured versus optimized poses, and calibrated versus optimized intrinsics.
  • PIVOT contributes a reusable multi-trajectory representation, dual-pose storage, calibrated and optimized intrinsics, a directed pose-space Chamfer distance, and three benchmark families.

II. RELATED WORK

Prior reconstruction work established strong novel-view synthesis methods, SfM-based camera estimation, and benchmark protocols, but commonly evaluates views within the training capture distribution. PIVOT instead emphasizes explicit trajectory diversity and deployment-oriented camera conditions.

  • Novel-view synthesis and neural reconstruction: NeRF and 3D Gaussian Splatting provide complementary neural and explicit scene representations for high-quality novel-view synthesis.
  • Camera pose estimation and Structure-from-Motion: COLMAP supplies SfM camera registration and sparse reconstruction, while PIVOT retains its optimized estimates alongside sensor-derived poses.
  • Reconstruction benchmarks and trajectory generalization: Established datasets such as LLFF and Mip-NeRF 360 measure rendering across several capture regimes, but test views commonly remain within the training capture distribution.
  • Calibration and pose robustness: Pose- and intrinsic-optimization methods address camera uncertainty offline, whereas PIVOT evaluates reconstruction using sensor-derived poses and reusable calibration conditions.
  • Reconstruction frameworks: PIVOT uses Nerfstudio as a benchmark backend and adds trajectory-aware export and evaluation for controlled pose, intrinsic, and viewpoint testing.
  • Scene capture design: Its scene protocol combines reconstruction-friendly paths with robot-like and extrapolation-oriented trajectories, including traversals, ascent, scattered views, and panorama stations.

B. Dual Pose Representation

PIVOT stores measured and COLMAP-optimized poses side by side, allowing translation and rotation sources to be varied independently. It likewise preserves physical and scene-optimized intrinsics for controlled calibration experiments.

  • Each processed frame can store both a sensor-derived measured camera-to-world pose and a COLMAP-optimized camera-to-world pose.
  • Measured poses use device metadata, including GPS position, flight attitude, and gimbal attitude, transformed into NED and OpenGL camera conventions.
  • COLMAP uses measured positions as soft priors, enabling SfM optimization while retaining a common spatial relationship with the measured trajectory.
  • The dual representation independently selects measured or optimized translation and rotation through OO, OM, MO, and MM configurations.
  • PIVOT retains physical/offline calibration and COLMAP-optimized per-scene intrinsics, with intrinsic selection independent of pose selection.

D. Processing Pipeline

PIVOT transforms raw multi-trajectory captures into processed scenes while preserving trajectory identity, camera alternatives, statistics, and evaluation metadata. Its directed pose-space Chamfer distance measures evaluation-trajectory coverage relative to training poses, but not viewing difficulty directly.

  • The processing pipeline preserves trajectory identity while converting raw video and photo captures into processed scenes.
  • Processing samples frames, writes metadata, computes measured poses, transforms coordinate conventions, runs COLMAP with soft priors, and retains optimized and calibrated camera parameters.
  • Directed pose Chamfer distance: The directed pose-space distance compares an evaluation trajectory A with a reference or training pose set B.
  • Directed pose Chamfer distance: The pose distance combines normalized translation and rotation components, with translation-only and rotation-only variants retained.
  • Directed pose Chamfer distance: Its directionality distinguishes how well training poses cover an evaluation trajectory from how well the evaluation trajectory covers training poses.
  • Directed pose Chamfer distance: During evaluation, the same formulation measures each evaluation trajectory against the complete training pose set.
  • Interpretation: Pose-space distance is a coverage descriptor rather than a direct measure of novel-view difficulty; experiments test its empirical association with quality degradation.

V. DATASET

PIVOT v1 provides five real-world drone-captured scenes with trajectory-preserving data, multiple pose and intrinsic sources, and reproducible processing and evaluation support.

  • PIVOT v1 contains five real-world scenes captured using a DJI Mini 4 Pro drone.
  • The dataset retains trajectory images, measured and COLMAP-optimized poses, calibrated and optimized intrinsics, and reconstruction statistics and assets.
  • Some trajectories are intentionally difficult for SfM, so PIVOT records both total and COLMAP-registered frames and treats registration rate as trajectory-design information.
  • Evaluation targets Nerfacto and Splatfacto and reports SSIM, PSNR, LPIPS, and directed pose Chamfer distance per trajectory.
  • The release freezes code commits, model configurations, iteration counts, random seeds, image settings, and exact trajectory selections for reproducibility.

A. Benchmark 1: Seen vs. Unseen Trajectories

Benchmark 1 evaluates whether reconstruction quality transfers from held-out views on training-represented trajectories to complete trajectories absent from training.

  • Benchmark 1 asks how quality changes between held-out frames on represented trajectories and complete trajectories not represented during training.
  • Training combines inward-orbit and traversal trajectories.
  • Seen trajectories use held-out frames from trajectories represented in training, whereas unseen trajectories are complete trajectories absent from the training set.
  • Per-trajectory evaluation reports image-quality metrics together with directed pose Chamfer distance to the training pose set.
  • The benchmark reports aggregate SSIM, LPIPS, and PSNR for Nerfacto and Splatfacto.

B. Trajectory Distance and Reconstruction Quality

PIVOT relates trajectory pose coverage to LPIPS and separately compares pose-source and intrinsic-source choices under controlled evaluation conditions.

  • Figure 3 relates directed normalized pose Chamfer distance to LPIPS for Nerfacto and Splatfacto across seen and unseen trajectories.
  • In Figure 3, seen trajectories are held-out views from represented training trajectories, while unseen trajectories are independently captured evaluation trajectories.
  • Benchmark 2 independently varies measured and optimized translation and rotation to examine their separate and combined effects on reconstruction degradation.
  • Pose-source combinations are denoted OO, OM, MO, and MM according to optimized or measured translation and rotation.
  • Benchmark 3 compares COLMAP-optimized scene intrinsics with fixed OpenCV calibration while holding pose source fixed.
  • Qualitative comparisons use identical ground-truth views across conditions for direct inspection of rendering changes.

IX. DISCUSSION

PIVOT distinguishes interpolation on represented trajectories from evaluation on structurally different paths, finding consistently worse unseen-trajectory quality while treating pose distance as descriptive rather than complete.

  • Held-out views on represented trajectories primarily probe interpolation, whereas independently captured trajectories probe viewpoints structurally departing from the training distribution.
  • Across all five scenes, unseen trajectories degrade relative to seen held-out views for both models on SSIM, LPIPS, and PSNR.
  • Nerfacto mean PSNR decreases from 19.54 to 16.47 dB in Church, 17.01 to 15.15 dB in Victorian Garden, and 19.80 to 16.82 dB in Village Street.
  • PIVOT uses directed pose Chamfer distance as a descriptive covariate rather than a complete predictor of novel-view difficulty.
  • Seen views generally have lower LPIPS near the training pose distribution, while unseen trajectories span broader pose distances and generally have higher LPIPS.
  • The pose-distance relationship is not strictly monotonic because similar pose-space distances can yield different rendering quality.The metric omits visibility, texture, occlusion, and scene content.

C. Sensitivity to Pose Source

PIVOT evaluates how pose-source choices affect reconstruction, while also testing intrinsic-source sensitivity and limitations of pose-only trajectory coverage metrics.

  • Pose-source sensitivity: Measured pose components can substantially reduce reconstruction quality relative to COLMAP-optimized poses, with effects varying across scenes and models.In Village Street, MM is 6.93 dB below OO for Nerfacto and 11.56 dB below OO for Splatfacto.
  • Intrinsic-source sensitivity: The intrinsic benchmark compares fixed physical calibration with per-scene COLMAP-optimized intrinsics.This contrasts a deployment-oriented reusable calibration setting with a favorable offline optimization setting.
  • Intrinsic-source sensitivity: Fixed OpenCV calibration reduces PSNR in every BM3 scene/model pair, by 1.75 dB to 9.33 dB.The observed effect is substantial but strongly scene- and model-dependent, and measures the specific calibration used in PIVOT v1.
  • Trajectory-distance limitation: Pose-space proximity does not necessarily imply similar scene visibility when occluding structures separate otherwise nearby cameras.A pose-only metric measures camera location and orientation, not the scene content visible from those viewpoints.
  • Trajectory-distance limitation: A geometry-aware trajectory metric could combine translation, rotation, and visibility overlap, while the current Chamfer distance remains a simple geometry-independent descriptor.The proposed extension could use projected reconstructed geometry and an Intersection-over-Union-based visibility measure.

B. Dataset Scale and Capture Platform

PIVOT v1 uses a controlled but narrow capture setup, while documenting calibration quality and registration behavior that constrain interpretation of its benchmarks.

  • Dataset scale and platform: PIVOT v1 contains five real-world scenes captured with one DJI Mini 4 Pro drone platform.This setup isolates target experimental variables but limits conclusions about other cameras, localization systems, environments, and motion platforms.
  • Calibration quality: The BM3 comparison depends on the quality of the independently estimated physical camera calibration.The PIVOT v1 calibration had approximately 4 pixels of reprojection error, so the intrinsic-source gap may partly reflect calibration quality.
  • SfM registration: Some trajectories may not register completely with SfM, and experiments requiring optimized poses must explicitly handle missing estimates.Configurations should report whether unregistered frames are dropped or evaluated using measured-pose fallbacks.
  • Geometry-aware metrics: The proposed geometry-aware trajectory metric would depend on reconstruction quality because sparse COLMAP points provide incomplete, potentially biased visibility.Future metrics should test sensitivity to the underlying geometric representation.

XI. REPRODUCIBILITY AND RELEASE

PIVOT releases code, data, containers, and an evaluation integration, supporting reproducible trajectory-aware reconstruction experiments.

  • Release artifacts: The source code is MIT-licensed, while the dataset uses the CC BY-NC 4.0 license.The project separates the core processing environment from the Nerfstudio integration environment.
  • Release artifacts: The release provides repository links for the PIVOT source, Nerfstudio integration, and dataset.These artifacts correspond to the v1.0.0 release locations listed by the paper.
  • Release artifacts: Versioned container images are provided for PIVOT processing and the Nerfstudio integration.The listed images are tagged 1.0.0.
  • Reproducible evaluation: PIVOT preserves measured and optimized poses, calibrated and optimized intrinsics, and explicit trajectory identity for controlled evaluation.Its directed pose Chamfer distance describes evaluation-trajectory coverage, while benchmark design distinguishes represented-trajectory held-out views from complete unseen trajectories.
  • Reproducible evaluation: Initial experiments consistently find gaps for unseen trajectories, measured poses, and alternative intrinsic sources, with magnitudes varying by scene and model.The paper positions these factors as explicit evaluation dimensions rather than universal performance predictors.
Loading 2608.25401v1…