PaperlayerView exact revision on arXiv

Source-linked AI summary

On-the-Fly3R: Towards Robust Online 3D Reconstruction with Feed-Forward 3R Models for Large-Scale UAV Scenarios

Zhe Shen, Liyuan Lou, Yifei Yu, Guanbo Wang, Quanjian Ji, Xin Wang, Zongqian Zhan

arXiv:2609.00923v1cs.CV

TL;DR

Large-scale UAV reconstruction must handle limited computational resources and unordered or weakly ordered imagery while avoiding spatially inconsistent local contexts and online failures. On-the-Fly3R uses retrieval-guided dynamic subsets, validation-rejection-retry, and progressive global integration to scale diverse feed-forward 3R models. Evaluations confirm processing thousands of images and square-kilometer UAV scenes within bounded GPU memory while matching offline global inference accuracy.

  • Problem

    Existing large-scale UAV reconstruction must operate under limited computational resources and unordered or weakly ordered images, where fixed local windows can combine spatially disjoint inputs.

  • Method

    On-the-Fly3R progressively reconstructs from a small seed by retrieving co-visible images into dynamic local subsets, aligning predictions to the global model, and validating, rejecting, and retrying unreliable integrations.

  • Results

    On-the-Fly3R processes thousands of images and square-kilometer UAV scenes within a bounded GPU memory budget while matching offline global inference accuracy across diverse benchmarks.

  • Takeaways & Limitations

    The framework scales diverse feed-forward 3R models to large-scale UAV imagery and suppresses cascading error propagation during progressive reconstruction.

Abstract

from arXiv · show

While feed-forward 3D reconstruction (3R) offers efficient end-to-end modeling, its application in large-scale UAV mapping is hindered by the prohibitive memory cost of Transformer attention. Current scalable streaming 3R methods assume temporally and spatially continuous inputs, rendering them ineffective for the weakly ordered or unordered image streams common in cross-strip UAV operations. To address this, we propose On-the-Fly3R, a training-free, progressive online 3D reconstruction framework for large-scale UAV images that upgrades various 3R backbones for large-scale UAV scenarios. Our method enables reconstruction from unordered inputs via retrieval-guided dynamic subset construction, which adaptively selects spatially relevant images. To further improve the robustness, a validation-rejection-retry mechanism is designed to guarantee global consistency, performing a pre-integration consistency check and automatically rejecting misaligned images and retrying with alternative subset. Finally, inspired by VSLAM, pose graph optimization based on the retrieval loop closure is employed to mitigate camera drift. Evaluations on several UAV benchmarks show that our On-the-Fly3R successfully scales various 3R models to over 5,000 images across square-kilometer UAV scenes, delivering substantially superior accuracy compared to several SOTA streaming 3R methods. Code is available at https://github.com/Sh1nZzz/On_the_Fly3R

I. INTRODUCTION

On-the-Fly3R addresses the memory and input-ordering barriers that limit feed-forward 3R models in large-scale UAV reconstruction. It combines retrieval-guided local contexts with validation and retry to support robust progressive reconstruction.

  • Transformer global attention makes scaling feed-forward 3R methods to large-scale UAV scenes computationally and memory intensive.
  • Existing chunk-based and streaming methods rely on strong spatiotemporal continuity and local overlap, which unordered UAV imagery often lacks.
  • The framework is training-free, progressive, and designed to scale feed-forward reconstruction to large-scale UAV scenarios.
  • On-the-Fly3R dynamically retrieves spatially relevant images to construct geometrically consistent local subsets for diverse 3R models.
  • Its validation–rejection–retry mechanism checks local reliability before global integration, filters unreliable references, and suppresses cascading error propagation.

II. RELATED WORKS

The related work spans traditional geometric pipelines, feed-forward 3R models, and scalable chunk-based or streaming approaches. On-the-Fly3R differs by retrieving geometrically relevant subsets and validating local results before integration.

  • Traditional 3D Reconstruction: Traditional SfM and SLAM recover structure incrementally through feature matching, geometric verification, optimization, tracking, or pose-graph methods.
  • Feed-forward 3D Reconstruction: Feed-forward 3R models predict camera poses and dense 3D structure end to end, with later methods improving matching, multi-view inference, ordering robustness, and generality.
  • Scalable Feed-forward 3D Reconstruction: Streaming methods maintain compact states or memories but depend on temporal order or sequential continuity for context construction.
  • Scalable Feed-forward 3D Reconstruction: Chunk-based methods partition sequences and align local reconstructions using overlap, geometric warping, optimization, or global context memory.
  • Scalable Feed-forward 3D Reconstruction: On-the-Fly3R replaces fixed chunks and temporal dependencies with retrieval-guided subsets and validation–rejection–retry to support unordered images and suppress error propagation.

III. METHODOLOGY

The methodology progressively builds a global reconstruction from retrieved local subsets while controlling memory and validating registration before fusion. It initializes a shared representation, stores compact geometric references, and estimates scene scale for adaptive validation.

  • Pipeline: The online pipeline initializes a global state from retrieval-connected seed images, processes incoming images, registers local reconstructions, validates them, and applies pose-graph optimization.
  • Initialization: A lightweight adapter unifies predictions from frozen 3R models into a standard representation containing camera transforms, intrinsics, dense point maps, and confidence maps.
  • Initialization: The initial prediction coordinate system becomes the global frame, and high-confidence points form the initial global point cloud.
  • Compact Representation: To maintain bounded memory, each accepted frame retains a compact geometric reference built from fixed numbers of high-confidence points sampled per image-grid cell.
  • Retrieval and Validation: Frozen SupScene descriptors support image retrieval, while characteristic scene spacing from adjacent high-confidence points determines a scale-adaptive validation threshold.

C. Retrieval-Guided Dynamic Subset Construction

Because online arrival order may lack spatial continuity, On-the-Fly3R retrieves historical context and dynamically builds geometrically compatible query and reference subsets for local 3R inference.

  • Fixed sliding windows can combine spatially disjoint images when online arrival order lacks spatial continuity.
  • Historical image descriptors provide shared context for newly arriving query frames.SupScene produces descriptors for candidate query frames, which are compared with accepted historical frames using cosine similarity.
  • The dynamic query subset expands by evaluating reference-overlap count and maximum visual similarity against existing queries.
  • A candidate enters the query subset only when it satisfies the defined compatibility condition with the current batch.The dominant reference set D(Q) is aggregated from the current query set.
  • Expansion stops when the next image is incompatible or the batch reaches Bmax, producing the final query set Qk.
  • Retrieval results are aggregated to prioritize highly similar historical frames supported by multiple queries, forming Rk for the final local 3R subset.The resulting local subset is then fed to the 3R model for unified reconstruction.

D. Confidence-Weighted Sim(3) Estimation

On-the-Fly3R registers each local reconstruction to the global map by estimating a confidence-weighted robust Sim(3) transformation from reference-frame correspondences.

  • Reference frames link the current local reconstruction with the historical global state.Stored pixel locations sample current local 3D points and pair them with historical global points.
  • Correspondence weights are defined as the geometric mean of the paired confidences.
  • The method estimates Sk = (σk, Rk, tk) ∈ Sim(3) to register the k-th local reconstruction to the global map.
  • Registration uses a confidence-weighted closed-form initialization followed by IRLS with a Huber robust loss to suppress outlier correspondences.

E. Validation-Retry Mechanism and Global Update

Before global integration, On-the-Fly3R validates local registration and retries with pruned references when consistency checks fail, rejecting unresolved local subsets.

  • Local registration can be unreliable because 3R predictions vary across contexts and retrieval may select visually similar but spatially unrelated references.
  • The framework enforces validation before global integration to prevent geometrically spurious Sim(3) estimates from propagating errors.
  • Reference-frame poses are transformed into the global frame and compared with stored poses using translation and rotation residuals.
  • Validation jointly checks residuals and whether the estimated scale σk is physically plausible, with translation thresholds scaled by scene spacing.
  • When validation fails, geometric pruning removes anomalous references using independent per-reference Sim(3) consistency analysis without rerunning 3R inference.
  • If geometric pruning fails, contextual pruning removes semantically incompatible references using register tokens and repeats inference, registration, and validation once.Only results passing the initial check or a retry stage are committed to the global state.
  • If both retry stages fail, the local subset is rejected and Mk remains Mk−1, isolating unreliable reconstruction from cascading errors.

F. Pose Graph Optimization

To reduce accumulated registration drift, On-the-Fly3R refines accepted camera poses with a robust pose graph containing retrieval, local-batch, and temporal-continuity constraints.

  • The pose graph contains nodes for all accepted frames, each associated with an optimizable camera pose Ti.
  • Retrieval-reference edges act as loop closures across large temporal gaps, while batch and continuity edges encode local and adjacent-frame constraints.
  • Continuity edges connect temporally adjacent arriving images only when retrieval association and motion plausibility checks are satisfied.
  • Edge weights are dynamically assigned from alignment robustness, using inlier ratio and residual magnitude.
  • Final camera poses are optimized on SE(3) with robust least squares using relative-pose observations, reliability information matrices, and a Huber kernel.

A. Datasets and Evaluation Metrics

The evaluation covers large-scale outdoor UAV scenes, indoor 7Scenes, pose and dense-reconstruction metrics, and standardized comparisons against scalable 3R baselines. On-the-Fly3R maintains broad coverage and improves pose and dense-reconstruction accuracy across these settings.

  • Metrics: Pose evaluation reports ATE and ARE for global discrepancies, plus RTE and RRE for relative translation and rotation errors.Dense reconstruction is also evaluated with point-cloud metrics.
  • Protocol: All baselines receive identical complete image sequences, and runtime is normalized per input image while peak GPU memory is measured across reconstruction.Returned-pose metrics are reported together with coverage because unreliable local updates may be rejected.
  • Outdoor results: 89%–96% pose coverage is maintained by all three On-the-Fly3R variants, with significantly lower errors across the outdoor scenes.FastVGGT and InfiniteVGGT encounter OOM on larger scenes, while TALO reaches only 21.24% coverage.
  • Indoor results: On 7Scenes, VGGT-Omega reduces scene-averaged ATE from 5.32 cm for MERG3R to 2.30 cm.VGGT-Omega achieves the lowest ATE across all seven scenes.
  • Dense reconstruction: On-the-Fly3R (Pi3x) achieves 3.85 m mean Chamfer Distance versus 16.41 m for MERG3R and 18.07 m for LingBot-Map.Its runtime is 0.72–0.81 s/frame, compared with 0.38 s/frame for LingBot-Map.

D. Comparison with Vanilla 3R Methods

The comparison tests whether progressive local updates preserve the accuracy of vanilla 3R joint inference under a shared 58-image input. On-the-Fly3R introduces only a marginal accuracy gap and often performs better.

  • Comparison setup: Under a 24 GB memory constraint, both approaches are compared using an identical 58-image input.This setting allows all evaluated backbones to complete native joint inference without OOM.
  • Results: On-the-Fly3R outperforms native inference in 7 out of 10 model-scene combinations.Table V marks the cases where progressive reconstruction exceeds vanilla joint inference.
  • Implication: Replacing global joint inference with progressive local updates introduces only a marginal accuracy gap while extending reconstruction to massive scenes.The authors attribute this preservation of geometric priors to retrieval-guided context and validation.

E. Ablation Study

The ablations show that validation-rejection-retry is the primary robustness driver, dynamic batching complements it, and pose graph optimization further reduces absolute trajectory error.

  • Experimental Setup: The ablation evaluates dynamic batching, validation-rejection-retry, and pose graph optimization using four pose RMSE metrics across five cases.VGGT-Omega is used as the backbone on HAV and Residence, with qualitative results shown in Fig. 4.
  • Impact of Dynamic Batching and V&R: On HAV, combining dynamic batching with validation-rejection-retry reduces ATE to 4.51 m, compared with 193.51 m using dynamic batching alone.Dynamic batching ensures spatial overlap, but does not by itself prevent catastrophic local failures.
  • Impact of Dynamic Batching and V&R: Without dynamic batching, validation-rejection-retry reduces HAV ATE to 30.94 m by rejecting inconsistent references.This identifies reliability gating as the primary contributor to robustness in the ablation.
  • Role of Pose Graph Optimization: Pose graph optimization further reduces HAV ATE to 3.64 m and Residence ATE from 2.34 to 1.25 m.On HAV, relative metrics increase slightly, while the longer Residence scene shows uniform metric improvements.
Loading 2609.00923v1…