PaperlayerView exact revision on arXiv

Source-linked AI summary

LonGaussian: Tracking Foundation Priors for Long-Context Feedforward Gaussian Reconstruction

Zhoufeng Xie, Guanjun Wu, Shuai Zhang, Xinggang Wang, Wenyu Liu

arXiv:2610.23250v1cs.CV

TL;DR

Long-video feed-forward reconstruction faces growing memory demands from jointly processing views and redundant scene representations from repeated surface observations. LonGaussian streams long-context geometry and consolidates Gaussians using multi-frame tracks, with filtering for cross-chunk artifacts. On DL3DV and RealEstate10K, it improves rendering quality while using substantially fewer Gaussians than feed-forward baselines.

  • Problem

    Long-video feed-forward reconstruction is constrained by the growing GPU memory demands of jointly processing views and redundant Gaussians accumulated from repeated surface observations.

    Source: §1 · p. 2

  • Method

    LonGaussian uses LoGeR’s streaming long-context geometry and track-guided Gaussian consolidation, followed by rendered-depth filtering of residual cross-chunk artifacts.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    On DL3DV and RealEstate10K, LonGaussian improves rendering quality while using substantially fewer Gaussians than feed-forward baselines.

    Source: §6 · p. 9

  • Takeaways & Limitations

    LonGaussian connects long-context geometry to compact, renderable Gaussian representations for long sequences.

    Source: §6 · p. 9

    LonGaussian assumes static scenes and reliable tracks, and its seven-chunk metrics do not measure full-scene global rendering fidelity.

    Source: §5 · p. 8

Abstract

from arXiv · show

Reconstructing high-quality, renderable 3D scenes from long videos is essential for robotic perception and embodied intelligence. Many feed-forward Gaussian reconstruction methods rapidly predict 3D scenes from a small set of images, but extending them to longer sequences presents two bottlenecks: jointly processing all views increases GPU memory requirements, while accumulating pixel-aligned predictions redundantly represents the same surfaces and increases storage and rendering costs. We present LonGaussian, a streaming feed-forward framework that reconstructs compact, globally renderable Gaussian scenes from long image sequences with known intrinsics and unknown camera poses. Built on a pretrained streaming 3D reconstruction backbone, LonGaussian decodes long-context geometric features into dense Gaussians. Within each chunk, multi-frame point tracks link repeated observations of the same surface and guide a learned merger to consolidate the corresponding Gaussian predictions. Rendered-depth consistency filtering further suppresses Gaussians from other chunks that incorrectly occlude locally reconstructed surfaces. On DL3DV and RealEstate10K benchmarks, LonGaussian improves novel-view rendering quality while using substantially fewer Gaussians than the evaluated feed-forward baselines.

1 Introduction

Long-video feed-forward reconstruction is limited by growing joint-view memory demands and redundant Gaussians from repeated surface observations. LonGaussian combines long-context streaming geometry with track-guided consolidation and filtering, improving rendering quality with fewer Gaussians on the reported benchmarks.

1 Introduction

  • Jointly processing input views makes computational and GPU memory requirements grow rapidly with sequence length, limiting long-video inference.Memory requirements may exceed GPU capacity.Source: §1 · p. 2
  • Accumulating pixel-aligned predictions retains overlapping Gaussians for repeated surface observations, increasing representation size and rasterization workload as sequences lengthen.Long-sequence feed-forward reconstruction therefore needs both streaming geometric processing and control of this redundancy.Source: §1 · p. 2
  • Point trajectories link observations of the same physical surface across frames and provide explicit correspondences for Gaussian grouping.Compared with voxel-occupancy or Euclidean-proximity strategies, track-based grouping offers a more direct and interpretable basis for consolidation.Source: §1 · p. 2
  • LonGaussian uses LoGeR’s chunk-wise hybrid memory for long-range geometric context and an independent tracker to group corresponding pixels into multi-frame trajectories.A learned merger consolidates their Gaussian predictions; rendered-depth consistency filtering suppresses residual cross-chunk floaters.Source: §1 · p. 2
  • LonGaussian extends LoGeR’s long-context geometric representation to compact, globally renderable Gaussian scenes.Its consolidation mechanism uses cross-frame track correspondences within chunks, with rendered-depth filtering for residual cross-chunk artifacts.Source: §1 · p. 3
  • LonGaussian achieves high-quality novel-view synthesis with substantially fewer Gaussians than existing feed-forward approaches on DL3DV and RealEstate10K.An ultra-long real-world sequence also demonstrates qualitative applicability at scale.Source: §1 · p. 3

2 Related Work

Related work spans long-context geometric reconstruction, feed-forward Gaussian prediction, and compact scene representations. LonGaussian builds on streaming geometry and uses multi-frame trajectories to consolidate repeated surface observations.

2.1 Feed-Forward and Long-Context 3D Reconstruction

  • Long-context models range from joint-view attention and recurrent state updates to LoGeR’s chunk-wise bidirectional reasoning with sliding-window attention and TTT fast weights.These models primarily predict point maps, depth, and camera parameters, leaving a bridge to compact, explicit Gaussian scenes missing.Source: §2.1 · p. 3

2.2 Feed-Forward Gaussian Reconstruction

  • Feed-forward Gaussian predictions are commonly pixel-aligned, so overlapping views can assign redundant or inconsistent primitives to the same surface as sequences grow.LonGaussian builds on LoGeR’s long-context backbone, separating long-context inference from explicit scene maintenance.Source: §2.2 · p. 3

2.3 Compact Gaussian Representations

  • Unlike compression or fusion strategies based on importance, proximity, or local matches, LonGaussian groups repeated surface observations through tracker-produced trajectories across a chunk.The related-work discussion frames resolving correspondence before consolidation as targeting the source of long-sequence redundancy while retaining feed-forward inference without per-scene refinement.Source: §2.3 · p. 3
  • Figure 3 depicts overlapping-chunk processing followed by track-guided consolidation, cross-chunk filtering, and far-field modeling.Snowflakes and flames mark frozen and trainable modules, respectively.Source: Figure 3 · p. 4

3 Method

LonGaussian inherits LoGeR’s chunk-wise long-context geometry and alignment, then decodes and consolidates Gaussian predictions into a shared scene. Track-guided merging, scene-level artifact handling, and staged image supervision support this pipeline, though shared coordinates do not guarantee drift-free alignment on long-range revisits.

3.1 Preliminaries

  • LoGeR processes overlapping chunks with local geometry and hybrid memory, and LonGaussian inherits its pretrained backbone, memory, and alignment to decode long-context features into a shared-coordinate Gaussian scene.The scene uses 3D Gaussian primitives with centers, rotations, anisotropic scales, opacity, and spherical-harmonic color coefficients.Source: §3.1 · p. 4

3.2 LonGaussian Framework

  • For each chunk, the framework decodes dense Gaussians, aligns them to the first chunk’s coordinate system, consolidates track-associated predictions, and applies scene-level filtering and far-field shell construction.Globally renderable means a single Gaussian representation in shared coordinates, not guaranteed drift-free alignment on long-range revisits.Source: §3.2 · p. 5

3.3 Track-Guided Gaussian Consolidation

  • Track-associated predictions are consolidated only when tracks meet observation-count, mean-opacity, and 3D-center agreement thresholds; each accepted track becomes one refined representative Gaussian.The merger starts from opacity-weighted attribute aggregation and refines it using pooled track observations, while uncovered predictions are retained.Source: §3.3 · p. 5

3.4 Scene-Level Artifact Handling

  • Rendered-depth comparisons across chunks identify persistent foreground floaters for suppression, while sky predictions are represented by angularly binned shell Gaussians.Removal requires multiple occlusion votes and minimum occlusion fractions among reliable comparisons; a chunk does not vote against its own predictions.Source: §3.4 · pp. 5–6

3.5 Training Strategy and Objectives

  • Training proceeds in stages with LoGeR and AllTracker frozen, first training the dense Gaussian branch and later training the merger and selected Gaussian attributes using context- and target-view image losses.The image terms measure RGB mean-squared error, LPIPS, and 1 − SSIM; the final stage includes depth filtering and the far-field shell.Source: §3.5 · p. 6

4 Experiments

LonGaussian outperforms the evaluated feed-forward methods on the main reconstruction comparison while using fewer Gaussians, and its single-scene scalability and component ablations reveal important scope and runtime trade-offs.

4.1 Experimental Setup

  • Evaluation compares offline and streaming baselines on DL3DV-140 and RealEstate10K test sequences, using up to 128 context frames and held-out target views.The two-pass unposed protocol estimates poses using context and target images, then constructs scenes from context images only; some methods were omitted because runnable official implementations were unavailable.Source: §4.1 · p. 6

4.2 Results

  • LonGaussian achieves the best PSNR and SSIM at every sequence length and uses the fewest Gaussians, though OF3GS slightly leads LPIPS on shortest-length RE10K.WorldMirror and AnySplat exceed the GPU memory budget even at the shortest evaluated length; LonGaussian also produces sharper, more coherent renderings.Source: Table 1 · p. 7, §4.2 · p. 7, Figure 4 · p. 7

4.3 Long-Sequence Scalability

  • On one scene, construction time and representation size grow approximately linearly while peak construction memory remains nearly constant; local rendering quality generally declines as the prefix expands.Quality uses seven-chunk rendering, whereas Gaussian count and construction resources cover the complete accumulated prefix.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 Ablation Studies

  • The full model reduces Gaussian count while improving PSNR and SSIM over dense accumulation, with a slight LPIPS trade-off; depth filtering yields the largest PSNR gain.Learned refinement improves quality over deterministic merging at similar representation size; removing the dense complement degrades quality, and omitting processing stages can shorten runtime despite retaining more Gaussians.Source: §4.4 · p. 8, Table 3 · p. 8

5 Discussion and Limitations

The discussion identifies assumptions and scale limits, illustrates two artifact treatments, and points to extensions for broader settings.

Limitations

  • The method assumes static scenes and reliable tracks; motion, geometric errors, fragmented tracks, or long-range pose errors may cause reconstruction errors or global rendering ghosting.Within-chunk tracking can duplicate surfaces revisited after long gaps, and depth voting and shell construction add overhead.Source: §5 · p. 8

Artifact Ablations

  • Figure 5 contrasts rendered-depth filtering for cross-chunk floaters with far-field modeling for unstable sky geometry.Colored boxes mark enlarged regions.Source: Figure 5 · p. 9

Future Directions

  • Future work could address dynamic scenes, cross-chunk association, uncertainty-aware filtering, and adaptive far-field representations.Source: §5 · p. 9

6 Conclusion

LonGaussian combines long-context geometry, track-guided consolidation, and artifact filtering to produce compact, renderable scenes. The paper also reports its AI-tool use, ethics scope, and resources intended to support reproduction.

6 Conclusion

  • LonGaussian combines LoGeR’s chunk-wise hybrid memory with track-guided consolidation and artifact filtering to reconstruct compact, globally renderable Gaussian scenes from long sequences.Experiments on DL3DV and RealEstate10K report improved rendering quality with substantially fewer Gaussians than feed-forward baselines.Source: §6 · p. 9

AI Use Statement

  • Generative AI tools assisted selected implementation components, translation, literature search, summarization, code debugging, and language refinement, but not methodology development or experimental interpretation.The authors state that they reviewed the AI-assisted work and take responsibility for the final content.Source: AI Use Statement · p. 9

Ethics Statement

  • The work studies static-scene reconstruction using existing benchmarks and pretrained models, and notes possible privacy misuse, inherited data biases, and lack of safety-critical evaluation.The paper says deployment should respect privacy, consent, copyright, dataset licenses, and applicable regulations.Source: Ethics Statement · pp. 9–10

Reproducibility Statement

  • The paper identifies its model, training, evaluation, implementation, and appendix materials as resources intended to support reproduction of reported results.Side-by-side rendering videos are available, and code will also be released.Source: Reproducibility Statement · p. 10

A Architecture and Consolidation Details

The section predicts dense per-pixel Gaussians, validates tracked observations before learned consolidation, and filters cross-chunk predictions using rendered-depth consistency.

A.1 Dense Gaussian Prediction Head

  • A DPT-style head fuses four decoder scales into full-resolution features and predicts Gaussian attributes for every context pixel.It predicts offsets, anisotropic scales, unit-quaternion rotations, opacity, and degree-2 spherical-harmonic coefficients; initialization discourages opaque or strongly displaced primitives.Source: §A.1 · p. 14

A.2 Track Validation and Ownership

  • Tracks qualify for consolidation with at least two observations, mean opacity of 0.05, and RMS center distance no greater than 0.3 backbone-coordinate units.The 0.3 threshold is absolute, not calibrated in meters or normalized by scene scale; the neighborhood gate adapts to local point spacing but does not change this threshold's scale dependence.Source: §A.2 · p. 14

A.3 Learned Track Merger

  • A learned merger pools accepted-track attributes and predicts bounded corrections to consolidate their Gaussian predictions.The pooled descriptor has 69 dimensions and is processed by a two-layer MLP with hidden dimension 128; opacity corrections use logit space and SH corrections are scaled by 0.25 tanh(·).Source: §A.3 · p. 14

B.1 Rendered-Depth Consistency Filtering

  • Cross-chunk Gaussians are removed only after reliable depth comparisons meet both occlusion-vote thresholds, while each chunk's own predictions are protected.Comparisons require sufficient opacity and limited depth spread; removal requires at least one occlusion vote and 30% thresholds for reliable-view and combined decision ratios.Source: §B.1 · p. 14

B.2 Filtering-Threshold Selection

Filtering thresholds are selected on a DL3DV training subset and fixed before test evaluation; the section also describes a far-field Gaussian shell for sky regions.

B.2 Filtering-Threshold Selection

  • Filtering improved mean PSNR, SSIM, and LPIPS across tested joint configurations, and the selected setting improved all 20 training-subset scenes over matched no-filter reconstructions.The setting uses a 1% relative depth gap, at least one occlusion vote, and 30% reliable-view and decision ratios; nearby candidates had similar mean PSNR, but parameters were varied jointly.Source: §B.2 · pp. 14–15, Table 4 · p. 15

B.3 Far-Field Gaussian Shell

  • Sky pixels are converted to rays and binned into azimuth–elevation cells, each producing an isotropic Gaussian on a shell at 50 times the scene radius.The shell center is the coordinate-wise median of camera centers; each Gaussian uses averaged direction and RGB color, opacity 0.98, and degree-zero SH.Source: §B.3 · p. 15

C Training Details

Training separates dense Gaussian prediction from learned-merger optimization, then adapts on RE10K and DL3DV sequences; evaluation samples context and held-out target views from prepared sequences.

C.1 Two-Stage Optimization

  • Training first optimizes the Gaussian decoder and heads, then trains the merger, followed by a short mixed-dataset adaptation.Stage I uses 8,590 eligible DL3DV scenes; Stage IIa trains the merger for 900 steps, and Stage IIb adapts for 300 steps on 150 RE10K sequences and 118 DL3DV scenes, with the feature decoder frozen.Source: §C.1 · p. 15, Table 5 · p. 16
  • Stage I uses AdamW at 10^-5, while Stage IIb uses 1.25 × 10^-5 for the merger and 2 × 10^-6 for the attribute head.Training uses four GPUs, gradient-norm clipping at 1.0, and image-loss weights λrgb = 1, λlpips = 0.25, λssim = 0.15, and λtgt = 1.Source: §C.1 · pp. 15–16
  • Two regularizers discourage needle-like Gaussians and large displacements from LoGeR point-map predictions.The anisotropy threshold is τaniso = 4.Source: §C.1 · p. 16

D.1 Dataset Preparation and View Sampling

  • Evaluation holds out every eighth frame as a target, giving 9, 13, or 18 target views for 64-, 96-, or 128-frame contexts.DL3DV uses downsampled images; RealEstate10K frames are center-cropped and resized to 280 × 490 with corresponding intrinsic updates.Source: §D.1 · p. 16

D.2 Two-Pass Unposed Evaluation

The evaluation separates camera estimation from context-only scene construction and compares methods under their pose-estimation protocols. In one scene, construction time and representation size scale approximately linearly across longer prefixes, but the snapshots use a fixed full-run pose trajectory.

D.2 Two-Pass Unposed Evaluation

  • The first pass estimates poses from context and target images; the second resets reconstruction state and builds the scene from context images only.Held-out target poses are aligned to the second-pass scene coordinates, but target appearance does not contribute scene primitives, tracking, filtering references, or shell samples.Source: §D.2 · p. 16

D.4 Single-Scene Long-Sequence Scalability

  • Construction time scales as T ∝ N^0.965 and representation size as G ∝ C^0.981 across six single-scene prefixes.The fits have R2 = 0.999 and R2 = 1.000, respectively; local rendering quality remains usable as sequence length grows, and peak memory remains bounded.Source: Figure 6 · p. 17, §D.4 · p. 18

D.3 Baseline Configuration

  • Baseline image metrics include each method's pose-estimation errors, and WorldMirror and AnySplat report OOM at the evaluated lengths.The numerical baselines predict target poses independently; ReCoSplat includes a locally adapted checkpoint rather than an official reproduction.Source: §D.3 · p. 17, §D.2 · p. 17

D.4 Single-Scene Long-Sequence Scalability

  • The scalability snapshots share the full run's first-pass pose trajectory, so they do not test strictly causal prefix-only camera estimation.Because longer prefixes also introduce new scene regions and different held-out views, the diagnostic is not a controlled fixed-content degradation curve.Source: §D.4 · p. 18

E Additional Analysis and Failure Cases

Additional comparisons show sharper structure and finer detail across scenes and sequence lengths, while complete-scene rendering can ghost; restricting views to seven neighboring chunks recovers much of the structure, with residual artifacts.

E Additional Analysis and Failure Cases

  • Restricting a target view to seven neighboring chunks recovers much of the scene structure lost to ghosting in complete-scene rendering.The improvement is consistent with interference from distant misregistered chunks, though local artifacts remain.Source: Figure 7 · p. 18
  • Reported Gaussian counts include track representatives, uncovered dense predictions, and far-field shell primitives; runtime includes tracking, two LoGeR passes, depth voting, and sky segmentation.Source: E Additional Analysis and Failure Cases · p. 18
  • Moving-object tracks can link observations without one shared 3D location, while difficult surfaces can fragment tracks and leave duplicated dense predictions.Geometric errors consistent within a LoGeR chunk may pass the track test; unstable local rendered depth can also cause thin foreground removal, and erroneous sky masks can replace finite content with shell Gaussians.Source: E Additional Analysis and Failure Cases · p. 18
  • LonGaussian preserves sharper structure and finer detail across scenes and sequence lengths in additional novel-view comparisons.Figure 8 presents these comparisons on DL3DV and RealEstate10K.Source: Figure 8 · p. 19

Related papers

Loading 2610.23250v1…