PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes

Fang Li, Hao Zhang, Narendra Ahuja

arXiv:2509.15123v2cs.CV

TL;DR

Camera estimation for dynamic scenes from casually captured RGB video remains challenging because common methods can require long runtimes or supervision beyond RGB. ROS-Cam combines sparse patch-wise tracking with uncertainty-aware joint optimization and a two-stage strategy. The authors report superior performance across five public datasets.

  • Problem

    Camera parameter optimization in casually collected dynamic-scene RGB videos remains challenging, while existing methods can be slow or depend on ground-truth motion masks or other priors typically unavailable in casual videos.

    Source: §1 · p. 2

  • Method

    ROS-Cam uses patch-wise tracking to derive sparse video trajectories, then jointly optimizes camera parameters and calibration-point uncertainty, with a two-stage optimization strategy.

    Source: §1 · pp. 2–3

  • Results

    The authors report superior performance in quantitative, qualitative, and ablation experiments across five public datasets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper presents RGB-only supervision as the basis for camera parameter optimization in dynamic scenes.

    Source: §1 · p. 3

    The method assumes constant focal length, and robustness in scenes dominated by large moving objects remains an unaddressed challenge.

    Source: §5 · p. 10

Abstract

from arXiv · show

Although COLMAP has long remained the predominant method for camera parameter optimization in static scenes, it is constrained by its lengthy runtime and reliance on ground truth (GT) motion masks for application to dynamic scenes. Many efforts attempted to improve it by incorporating more priors as supervision such as GT focal length, motion masks, 3D point clouds, camera poses, and metric depth, which, however, are typically unavailable in casually captured RGB videos. In this paper, we propose a novel method for more accurate and efficient camera parameter optimization in dynamic scenes solely supervised by a single RGB video, dubbed ROS-Cam. Our method consists of three key components: (1) Patch-wise Tracking Filters, to establish robust and maximally sparse hinge-like relations across the RGB video. (2) Outlier-aware Joint Optimization, for efficient camera parameter optimization by adaptive down-weighting of moving outliers, without reliance on motion priors. (3) A Two-stage Optimization Strategy, to enhance stability and optimization speed by a trade-off between the Softplus limits and convex minima in losses. We visually and numerically evaluate our camera estimates. To further validate accuracy, we feed the camera estimates into a 4D reconstruction method and assess the resulting 3D scenes, and rendered 2D RGB and depth maps. We perform experiments on 4 real-world datasets (NeRF-DS, DAVIS, iPhone, and TUM-dynamics) and 1 synthetic dataset (MPI-Sintel), demonstrating that our method estimates camera parameters more efficiently and accurately with a single RGB video as the only supervision.

1 Introduction

The introduction frames accurate, efficient camera estimation from casually captured dynamic-scene RGB video as an open challenge, since existing options rely on lengthy runtimes, motion masks, or additional supervision. ROS-Cam combines sparse tracking, outlier-aware joint optimization, and a two-stage strategy; the paper reports superior performance across five public datasets.

1 Introduction

  • Camera parameter optimization from casually collected RGB videos of dynamic scenes remains challenging, while COLMAP is slow and requires ground-truth motion masks for dynamic scenes.Other approaches add priors such as focal length, metric depth, point clouds, camera poses, or motion masks, which are typically unavailable in casual videos.Source: §1 · p. 2
  • Existing RGB-only methods rely on multiple pre-trained dense prediction models, can degrade when one fails, and cannot adaptively exclude moving outliers without motion supervision.The introduction also identifies their high computational latency as a source of lengthy runtimes.Source: §1 · p. 2
  • ROS-Cam uses patch-wise tracking filters built on a pre-trained point-tracking model to create robust, maximally sparse hinge-like relations across video frames.The formulation aims to avoid inaccurate trajectories and latency associated with noisy dense-prediction pseudo-supervision.Source: §1 · p. 2
  • The method jointly optimizes calibration points, camera parameters, and Cauchy-modeled uncertainty to down-weight moving-outlier effects without motion priors.Uncertainty is associated with sparse 3D calibration points rather than 2D pixels, reducing the number of learnable parameters and runtime relative to casualSAM and LEAP-VO.Source: §1 · p. 2
  • Experiments on five public datasets demonstrate the method’s superior performance, with quantitative, qualitative, and ablation evaluations.The datasets are NeRF-DS, DAVIS, iPhone, MPI-Sintel, and TUM-dynamics.Source: §1 · p. 3
  • The paper presents ROS-Cam as an RGB-only method with three components: patch-wise tracking filters, outlier-aware joint optimization, and two-stage optimization.The two-stage strategy is described as addressing local-minimum risk and accelerating and stabilizing optimization.Source: §1 · pp. 2–3

2 Related Works

Related work spans multiple scene representations and camera-estimation supervision regimes. The paper contrasts RGB-only approaches with methods that use additional ground-truth priors or large-scale supervised training, and notes remaining challenges for dynamic scenes.

Dynamic Scene Reconstruction/Novel View Synthesis (NVS)

  • Dynamic-scene reconstruction uses planar, mesh, point-cloud, neural-field, and Gaussian representations, while NeRF-based methods have long training times and 3DGS supports near-real-time dynamic-scene synthesis.Both NeRF- and 3DGS-based methods heavily rely on COLw/mask for camera parameter estimation.Source: §2 · p. 3

Camera Parameter Optimization

  • Camera-estimation methods vary in supervision: some use ground-truth focal length, depth, or motion priors, while feed-forward DUSt3R-based methods use ground-truth point clouds and poses.The DUSt3R-based approaches described require several days of high-end GPU training, and metric 3D supervision is relatively scarce, with domain gaps on unseen data.Source: §2 · p. 3
  • RGB-only methods include Robust-CVD, casualSAM, and ROS-Cam, but the paper reports their performance as significantly worse than ROS-Cam’s.Other RGB-only approaches discussed cannot handle moving objects in dynamic scenes, while motion-supervised methods address them using ground-truth motion information.Source: §2 · p. 3

3 Method

ROS-Cam extracts sparse tracking trajectories from RGB frames and jointly estimates calibration points and camera parameters while modeling motion-related uncertainty. Its filters select and prune tracks, and a two-stage schedule is intended to stabilize and accelerate optimization.

3 Method

  • Given RGB frames, the method extracts sparse tracking trajectories as pseudo-supervision and jointly optimizes calibration points, focal length, camera rotations and translations, and motion-related uncertainty.The two-stage strategy uses Softplus behavior and convex-loss minima to target stability and optimization speed.Source: §3 · pp. 3–4

3.1 Patch-wise Tracking Filters

  • Texture-rich patches supply candidate tracks; gradient, visibility, and distribution filters retain strong, visible points while limiting duplicate tracks within a patch.The retained points and trajectory indices act as pseudo-supervision for joint optimization.Source: §3.1 · pp. 4–5

3.2 Outlier-aware Joint Optimization

  • The joint optimization projects calibration points into frames and learns per-point uncertainty with a Cauchy model to mitigate moving outliers without ground-truth motion priors.The Cauchy distribution is described as handling heavy tails better than a Gaussian; the figure contrasts lower uncertainty for static samples with the opposite behavior for dynamic samples.Source: §3.2 · p. 5, Figure 3 · p. 5
  • The runtime figure shows ROS-Cam’s runtime growing almost linearly with frame count, compared with roughly exponential growth for COLw/omask.The complete runtime comparison is referred to in Table 3 and supplementary tables.Source: Figure 4 · p. 5

3.3 Two-stage Optimization Strategy

  • Stage 1 fixes the raw uncertainty parameter and optimizes points and camera parameters for rapid convergence; Stage 2 initializes uncertainty from Stage 1’s ACP error and optimizes jointly for stability.The strategy is motivated by Softplus limits and analytical minima of the inner convex term in the Cauchy loss.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate ROS-Cam’s runtime, camera-pose accuracy, and novel-view synthesis using shared reconstruction setups where ground-truth camera parameters are unavailable. Across the reported evaluations, it is efficient and performs strongly against RGB-only methods and methods using additional priors, while ablations identify contributions from its filters and optimization strategy.

4.1 Implementation Details

  • For datasets without GT camera parameters, methods’ estimates are evaluated through 4DGS using NVS metrics; datasets with GT parameters are evaluated directly using camera-pose metrics.The experiments use one NVIDIA A100 40GB GPU, with 200 Stage 1 and 50 Stage 2 iterations.Source: §4 · p. 6, §4.1 · p. 6, §4.4 · p. 8

4.4 NVS Evaluation

  • ROS-Cam is best on NeRF-DS and DAVIS, and is the best RGB-only method on iPhone, where it occasionally surpasses LiDAR-based Record3D.For iPhone, the reported Record3D camera parameters are occasionally unreliable.Source: Table 2 · p. 7, §4.4 · p. 8, Table 6 · p. 8

4.3 Camera Pose Evaluation

  • ROS-Cam achieves the best overall performance among RGB-only methods on TUM-dynamics and is better overall than methods supervised by additional GT priors.On MPI-Sintel, it also outperforms casualSAM; COLw/o mask is excluded because it fails.Source: Table 5 · p. 6, Table 4 · p. 7, §4.3 · p. 8

4.2 Time Efficiency Evaluation

  • The method’s average runtime on NeRF-DS, DAVIS, and iPhone is 55%, 11%, and 8%, respectively, of the second-fastest methods’ runtime.Its runtime grows linearly at about 1/800 hours per frame, while COLw/o mask grows roughly exponentially.Source: §4.2 · p. 7

4.3 Camera Pose Evaluation

  • On MPI-Sintel, estimated camera trajectories almost perfectly align with ground truth, qualitatively supporting the camera-pose results.The trajectories are shown alongside ground truth in Figure 6.Source: Figure 6 · p. 7, §4.3 · p. 8

4.4 NVS Evaluation

  • Removing any tracking filter weakens video-wide relations, while removing Γ, EACP, or the two-stage strategy harms results in the presence of outliers.The two-stage strategy’s omission causes a performance drop at the same iteration count; additional iterations are necessary but not sufficient for comparable results.Source: §4.4 · pp. 9–10, Table 7 · p. 9

5 Conclusion and Limitation

The conclusion presents an RGB-only method for estimating per-frame camera parameters in casually captured dynamic scenes without additional ground-truth supervision. The section also describes 4D Gaussian Splatting representations used to evaluate camera estimates through novel-view synthesis.

5 Conclusion and Limitation

  • ROS-Cam estimates per-frame camera parameters from casually collected dynamic-scene videos without additional ground-truth supervision beyond RGB.The authors say it may serve as a step toward high-fidelity reconstruction from casually captured videos.Source: §5 · p. 10
  • The method assumes constant focal length, leaving RGB-only calibration for zooming videos open; scenes dominated by large moving objects also challenge its tracking relations.In such scenes, most extracted trajectories may be outliers.Source: §5 · p. 10

A Dynamic Scene Optimization

  • Each 3D Gaussian is parameterized by its center, quaternion, opacity, scale, and color; its world covariance is transformed into camera coordinates.The covariance transformation uses the view transformation and the Jacobian of the affine parts of the projective transformation.Source: A Dynamic Scene Optimization · p. 15
  • Rendered color uses spherical-harmonic coefficients and density associated with the Gaussian points.Source: A Dynamic Scene Optimization · p. 15
  • For camera-estimate evaluation, 4DGS learns canonical Gaussian representations and deformation offsets, with image-to-rendering L1 supervision on the deformations.The authors compare novel-view-synthesis performance using the same scene-optimization method with different camera estimates.Source: A Dynamic Scene Optimization · p. 15

B Derivation of Cauchy Negative-log-likelihood

The derivation turns the Cauchy probability model for calibration-point uncertainty into an optimization loss by maximizing likelihood, then minimizing negative log-likelihood.

C Datasets

The evaluation spans five public datasets with varied scene and motion characteristics, using dataset-specific splits, frame selections, and evaluation setups.

  • Videos across the five datasets vary in camera and object motion and texture, range from about 50 to 900 frames, and use alternating adjacent frames for NVS train and test.For pose evaluation, the setup follows Cut3r and Monst3r on TUM-dynamics and evaluates all MPI-Sintel videos.Source: C Datasets · p. 16
  • NeRF-DS contains seven 400–800-frame indoor videos with moving specular objects, mixed-texture backgrounds, large scene and camera motion, and some blur.The experiments use the highest available 480 × 270 resolution; GT motion masks are human-labeled and camera parameters come from COLw/ mask.Source: C Datasets · p. 16
  • DAVIS includes 40 short wild-scene videos, while the experiments use 21 videos with large camera and object movements after excluding some fixed-camera or variable-focal-length cases.The input resolution is 854 × 480.Source: C Datasets · p. 16
  • The iPhone dataset comprises 14 casually recorded indoor and outdoor videos with rapid object motion and significant camera movement, without GT motion masks.The experiments use all videos at 720 × 960 resolution and compare against Record3D camera estimates.Source: C Datasets · p. 16
  • MPI-Sintel provides GT camera parameters for 18 short synthetic videos with large object motion, and this paper evaluates all videos rather than the 14 used by many prior works.The authors note a domain gap between synthetic MPI-Sintel and real-world scenarios.Source: C Datasets · p. 16
  • TUM-dynamics contains eight long, blurry real-world indoor videos with significant depth of field and large camera and object movements.The setup samples the first 90 frames at temporal stride 3 to save computation.Source: C Datasets · p. 16

D Evaluation Metrics

The evaluation combines novel-view-synthesis metrics on datasets without GT camera parameters, pose metrics where GT poses are available, and runtime comparisons.

D Evaluation Metrics

  • NVS evaluation uses PSNR, SSIM, and LPIPS on NeRF-DS, DAVIS, and iPhone; pose evaluation uses ATE, RPE trans, and RPE rot on MPI-Sintel and TUM-dynamics.Time evaluation is also used to demonstrate time efficiency.Source: D Evaluation Metrics · p. 16

D.1 PSNR & SSIM & LPIPS

  • PSNR measures the ratio of maximum possible signal power to corrupting-noise power, using mean squared error between rendered and original images.The image maximum pixel value is denoted MAX.Source: §D.1 · p. 17
  • SSIM compares luminance, contrast, and structure, is intended to match human perception more closely than PSNR, and ranges from -1 to 1, with 1 denoting perfection.Source: §D.1 · p. 17
  • LPIPS measures perceptual similarity by comparing deep-network feature activations of image patches; this paper uses VGG-based networks.Source: §D.1 · p. 17

D.2 ATE & RPE trans & RPE rot

  • ATE measures global trajectory error by aligning the estimated trajectory to ground truth and computing corresponding-point Euclidean distances.Source: §D.2 · p. 17
  • RPE trans measures translational error between poses over a time or distance interval, emphasizing local relative-motion accuracy rather than whole-trajectory error.Source: §D.2 · p. 17
  • RPE rot measures orientation error between estimated and ground-truth poses over short sequences, typically in degrees or radians.Source: §D.2 · p. 17

E.1 Quantitative Results

ROS-Cam has the shortest runtime in the reported comparisons, with a Plate-video exception where faster COLMAP runs result from failure. Runtime differences become more pronounced on the datasets’ videos containing over 50 frames.

E.1 Quantitative Results

  • ROS-Cam has the shortest runtime on the reported NeRF-DS, DAVIS, and iPhone comparisons, except for COLMAP on Plate, where it fails.The Plate runs converge quickly to a local minimum after failure.Source: §E.1.1 · p. 17
  • Runtime differences become more pronounced on NeRF-DS, DAVIS, and iPhone videos containing over 50 frames.Source: §E.1.1 · p. 17

E.2.1 Trajectories from Patch-wise Tracking Filters as Pseudo-supervision

Patch-wise filters improve trajectory reliability in low-texture regions and add reliable tracks when too few remain. Downstream renderings and reconstructed fields compare favorably with other RGB-only methods, with iPhone fields comparable to or better than LiDAR-based Record3D.

E.2.1 Trajectories from Patch-wise Tracking Filters as Pseudo-supervision

E.2.2 NVS

E.2.3 Optimized 3D Gaussian Fields

  • On iPhone, reconstructed 3D Gaussian fields are comparable to or better than LiDAR-based Record3D, particularly in scenes with significant motion.The iPhone dataset is described as especially challenging because of large camera and object movements.Source: §E.2.3 · p. 19, Figure 19 · p. 26
Loading 2509.15123v2…