PaperlayerView exact revision on arXiv

Source-linked AI summary

On-the-Fly3R: Towards Robust Online 3D Reconstruction with Feed-Forward 3R Models for Large-Scale UAV Scenarios

Zhe Shen, Liyuan Lou, Yifei Yu, Guanbo Wang, Quanjian Ji, Xin Wang, Zongqian Zhan

arXiv:2609.00923v1cs.CV

TL;DR

Large-scale UAV reconstruction must handle limited computational resources and unordered or weakly ordered imagery while avoiding spatially inconsistent local contexts and online failures. On-the-Fly3R uses retrieval-guided dynamic subsets, validation-rejection-retry, and progressive global integration to scale diverse feed-forward 3R models. Evaluations confirm processing thousands of images and square-kilometer UAV scenes within bounded GPU memory while matching offline global inference accuracy.

  • Problem

    Existing large-scale UAV reconstruction must operate under limited computational resources and unordered or weakly ordered images, where fixed local windows can combine spatially disjoint inputs.

    Source: §I · p. 2, C. Retrieval-Guided Dynamic Subset… · p. 3

  • Method

    On-the-Fly3R progressively reconstructs from a small seed by retrieving co-visible images into dynamic local subsets, aligning predictions to the global model, and validating, rejecting, and retrying unreliable integrations.

    Source: §I · p. 2, E. Validation-Retry Mechanism and… · p. 5

  • Results

    On-the-Fly3R processes thousands of images and square-kilometer UAV scenes within a bounded GPU memory budget while matching offline global inference accuracy across diverse benchmarks.

    Source: §V · p. 8

  • Takeaways & Limitations

    The framework scales diverse feed-forward 3R models to large-scale UAV imagery and suppresses cascading error propagation during progressive reconstruction.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

While feed-forward 3D reconstruction (3R) offers efficient end-to-end modeling, its application in large-scale UAV mapping is hindered by the prohibitive memory cost of Transformer attention. Current scalable streaming 3R methods assume temporally and spatially continuous inputs, rendering them ineffective for the weakly ordered or unordered image streams common in cross-strip UAV operations. To address this, we propose On-the-Fly3R, a training-free, progressive online 3D reconstruction framework for large-scale UAV images that upgrades various 3R backbones for large-scale UAV scenarios. Our method enables reconstruction from unordered inputs via retrieval-guided dynamic subset construction, which adaptively selects spatially relevant images. To further improve the robustness, a validation-rejection-retry mechanism is designed to guarantee global consistency, performing a pre-integration consistency check and automatically rejecting misaligned images and retrying with alternative subset. Finally, inspired by VSLAM, pose graph optimization based on the retrieval loop closure is employed to mitigate camera drift. Evaluations on several UAV benchmarks show that our On-the-Fly3R successfully scales various 3R models to over 5,000 images across square-kilometer UAV scenes, delivering substantially superior accuracy compared to several SOTA streaming 3R methods. Code is available at https://github.com/Sh1nZzz/On_the_Fly3R

I. INTRODUCTION

On-the-Fly3R addresses the memory and input-ordering barriers that limit feed-forward 3R models in large-scale UAV reconstruction. It combines retrieval-guided local contexts with validation and retry to support robust progressive reconstruction.

  • Transformer global attention makes scaling feed-forward 3R methods to large-scale UAV scenes computationally and memory intensive.Source: §I · pp. 1–2
  • Existing chunk-based and streaming methods rely on strong spatiotemporal continuity and local overlap, which unordered UAV imagery often lacks.Source: §I · pp. 1–2
  • The framework is training-free, progressive, and designed to scale feed-forward reconstruction to large-scale UAV scenarios.Source: §I · p. 2
  • On-the-Fly3R dynamically retrieves spatially relevant images to construct geometrically consistent local subsets for diverse 3R models.Source: §I · p. 2
  • Its validation–rejection–retry mechanism checks local reliability before global integration, filters unreliable references, and suppresses cascading error propagation.Source: §I · p. 2

II. RELATED WORKS

The related work spans traditional geometric pipelines, feed-forward 3R models, and scalable chunk-based or streaming approaches. On-the-Fly3R differs by retrieving geometrically relevant subsets and validating local results before integration.

Traditional 3D Reconstruction

  • Traditional SfM and SLAM recover structure incrementally through feature matching, geometric verification, optimization, tracking, or pose-graph methods.Source: A. Traditional 3D Reconstruction · p. 2

Feed-forward 3D Reconstruction

Scalable Feed-forward 3D Reconstruction

  • Streaming methods maintain compact states or memories but depend on temporal order or sequential continuity for context construction.Source: C. Scalable Feed-forward 3D… · pp. 2–3
  • Chunk-based methods partition sequences and align local reconstructions using overlap, geometric warping, optimization, or global context memory.Source: C. Scalable Feed-forward 3D… · p. 3
  • On-the-Fly3R replaces fixed chunks and temporal dependencies with retrieval-guided subsets and validation–rejection–retry to support unordered images and suppress error propagation.Source: C. Scalable Feed-forward 3D… · p. 3

III. METHODOLOGY

The methodology progressively builds a global reconstruction from retrieved local subsets while controlling memory and validating registration before fusion. It initializes a shared representation, stores compact geometric references, and estimates scene scale for adaptive validation.

Pipeline

  • The online pipeline initializes a global state from retrieval-connected seed images, processes incoming images, registers local reconstructions, validates them, and applies pose-graph optimization.Source: A. Overview · p. 3, B. Initialization · p. 3

Initialization

  • A lightweight adapter unifies predictions from frozen 3R models into a standard representation containing camera transforms, intrinsics, dense point maps, and confidence maps.Source: B. Initialization · p. 3
  • The initial prediction coordinate system becomes the global frame, and high-confidence points form the initial global point cloud.Source: B. Initialization · p. 3

Compact Representation

  • To maintain bounded memory, each accepted frame retains a compact geometric reference built from fixed numbers of high-confidence points sampled per image-grid cell.Source: B. Initialization · p. 3

Retrieval and Validation

  • Frozen SupScene descriptors support image retrieval, while characteristic scene spacing from adjacent high-confidence points determines a scale-adaptive validation threshold.Source: B. Initialization · p. 3

C. Retrieval-Guided Dynamic Subset Construction

Because online arrival order may lack spatial continuity, On-the-Fly3R retrieves historical context and dynamically builds geometrically compatible query and reference subsets for local 3R inference.

D. Confidence-Weighted Sim(3) Estimation

On-the-Fly3R registers each local reconstruction to the global map by estimating a confidence-weighted robust Sim(3) transformation from reference-frame correspondences.

E. Validation-Retry Mechanism and Global Update

Before global integration, On-the-Fly3R validates local registration and retries with pruned references when consistency checks fail, rejecting unresolved local subsets.

F. Pose Graph Optimization

To reduce accumulated registration drift, On-the-Fly3R refines accepted camera poses with a robust pose graph containing retrieval, local-batch, and temporal-continuity constraints.

  • The pose graph contains nodes for all accepted frames, each associated with an optimizable camera pose Ti.Source: F. Pose Graph Optimization · p. 5
  • Retrieval-reference edges act as loop closures across large temporal gaps, while batch and continuity edges encode local and adjacent-frame constraints.Source: F. Pose Graph Optimization · p. 5
  • Continuity edges connect temporally adjacent arriving images only when retrieval association and motion plausibility checks are satisfied.Source: F. Pose Graph Optimization · p. 5
  • Edge weights are dynamically assigned from alignment robustness, using inlier ratio and residual magnitude.Source: F. Pose Graph Optimization · p. 5
  • Final camera poses are optimized on SE(3) with robust least squares using relative-pose observations, reliability information matrices, and a Huber kernel.Source: F. Pose Graph Optimization · p. 5

A. Datasets and Evaluation Metrics

The evaluation covers large-scale outdoor UAV scenes, indoor 7Scenes, pose and dense-reconstruction metrics, and standardized comparisons against scalable 3R baselines. On-the-Fly3R maintains broad coverage and improves pose and dense-reconstruction accuracy across these settings.

Metrics

  • Pose evaluation reports ATE and ARE for global discrepancies, plus RTE and RRE for relative translation and rotation errors.Dense reconstruction is also evaluated with point-cloud metrics.Source: A. Datasets and Evaluation Metrics · p. 5

Protocol

Outdoor results

  • 89%–96% pose coverage is maintained by all three On-the-Fly3R variants, with significantly lower errors across the outdoor scenes.FastVGGT and InfiniteVGGT encounter OOM on larger scenes, while TALO reaches only 21.24% coverage.Source: C. Comparison with other SOTA 3R Methods · p. 6

Indoor results

Dense reconstruction

D. Comparison with Vanilla 3R Methods

The comparison tests whether progressive local updates preserve the accuracy of vanilla 3R joint inference under a shared 58-image input. On-the-Fly3R introduces only a marginal accuracy gap and often performs better.

Comparison setup

Results

Implication

  • Replacing global joint inference with progressive local updates introduces only a marginal accuracy gap while extending reconstruction to massive scenes.The authors attribute this preservation of geometric priors to retrieval-guided context and validation.Source: D. Comparison with Vanilla 3R Methods · p. 7

E. Ablation Study

The ablations show that validation-rejection-retry is the primary robustness driver, dynamic batching complements it, and pose graph optimization further reduces absolute trajectory error.

Experimental Setup

  • The ablation evaluates dynamic batching, validation-rejection-retry, and pose graph optimization using four pose RMSE metrics across five cases.VGGT-Omega is used as the backbone on HAV and Residence, with qualitative results shown in Fig. 4.Source: E. Ablation Study · p. 7

Impact of Dynamic Batching and V&R

  • On HAV, combining dynamic batching with validation-rejection-retry reduces ATE to 4.51 m, compared with 193.51 m using dynamic batching alone.Dynamic batching ensures spatial overlap, but does not by itself prevent catastrophic local failures.Source: E. Ablation Study · p. 7
  • Without dynamic batching, validation-rejection-retry reduces HAV ATE to 30.94 m by rejecting inconsistent references.This identifies reliability gating as the primary contributor to robustness in the ablation.Source: E. Ablation Study · p. 7

Role of Pose Graph Optimization

  • Pose graph optimization further reduces HAV ATE to 3.64 m and Residence ATE from 2.34 to 1.25 m.On HAV, relative metrics increase slightly, while the longer Residence scene shows uniform metric improvements.Source: E. Ablation Study · p. 7

Related papers

Loading 2609.00923v1…