PaperlayerView exact revision on arXiv

Source-linked AI summary

Lite3R: A Model-Agnostic Framework for Efficient Feed-Forward 3D Reconstruction

Haoyu Zhang, Zeyu Zhang, Zedong Zhou, Yang Zhao, Hao Tang

arXiv:2605.11354v1cs.CV

TL;DR

Transformer-based 3D reconstruction faces high dense-attention costs and instability when geometry-sensitive models run at low precision. Lite3R addresses both issues with Sparse Linear Attention and parameter-efficient FP8-aware QAT with partial attention distillation, reducing latency and memory while preserving competitive reconstruction quality overall.

  • Problem

    Transformer-based 3D reconstruction becomes increasingly costly as backbones and inputs scale, while low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency.

  • Method

    Lite3R uses a teacher–student framework combining Sparse Linear Attention, parameter-efficient FP8-aware QAT, and partial attention distillation while freezing most pretrained backbone parameters.

  • Results

    Lite3R reduces latency by 1.76–1.97× and memory footprint by 2.32–2.71× across VGGT and DA3-Large experiments while maintaining competitive depth, pose, and 3D reconstruction quality overall.

  • Takeaways & Limitations

    Lite3R provides a practical algorithm–system co-design approach for deployment-oriented, low-precision transformer-based 3D reconstruction.

  • Takeaways & Limitations

    Geometry-sensitive components such as LayerNorm, positional encoding, RoPE, and selected nonlinear operators remain in higher precision when needed.

Abstract

from arXiv · show

Transformer-based 3D reconstruction has emerged as a powerful paradigm for recovering geometry and appearance from multi-view observations, offering strong performance across challenging visual conditions. As these models scale to larger backbones and higher-resolution inputs, improving their efficiency becomes increasingly important for practical deployment. However, modern 3D transformer pipelines face two coupled challenges: dense multi-view attention creates substantial token-mixing overhead, and low-precision execution can destabilize geometry-sensitive representations and degrade depth, pose, and 3D consistency. To address the first challenge, we propose Lite3R, a model-agnostic teacher-student framework that replaces dense attention with Sparse Linear Attention to preserve important geometric interactions while reducing attention cost. To address the second challenge, we introduce a parameter-efficient FP8-aware quantization-aware training (FP8-aware QAT) strategy with partial attention distillation, which freezes the vast majority of pretrained backbone parameters and trains only lightweight linear-branch projection layers, enabling stable low-precision deployment while retaining pretrained geometric priors. We further evaluate Lite3R on two representative backbones, VGGT and DA3-Large, over BlendedMVS and DTU64, showing that it substantially reduces latency (1.7-2.0x) and memory usage (1.9-2.4x) while preserving competitive reconstruction quality overall. These results demonstrate that Lite3R provides an effective algorithm-system co-design approach for practical transformer-based 3D reconstruction. Code: https://github.com/AIGeeksGroup/Lite3R. Website: https://aigeeksgroup.github.io/Lite3R.

1 Introduction

Lite3R targets the efficiency challenges of transformer-based 3D reconstruction by reducing dense-attention cost and enabling practical low-precision deployment. It combines sparse attention, parameter-efficient FP8-aware training, and partial distillation while maintaining competitive reconstruction quality.

  • Transformer-based 3D reconstruction delivers strong geometry, pose, and 3D consistency, but scaling increases the need for efficient deployment.
  • Efficient systems should preserve important cross-view interactions rather than relying on naive pruning or uniform compression.
  • Lite3R replaces dense attention with Sparse Linear Attention in a model-agnostic teacher–student framework to reduce attention cost and memory.
  • Lite3R’s FP8-aware QAT freezes most pretrained backbone parameters and trains lightweight linear-branch projection layers for low-precision deployment.
  • Experiments on VGGT and DA3-Large across BlendedMVS and DTU64 show reduced latency and memory while maintaining competitive depth, pose, and 3D reconstruction quality.Reported reductions are 1.7–2.0× in latency and 1.9–2.4× in memory footprint.

2 Related Work

Related work improves 3D reconstruction efficiency through redesigned attention, practical adaptation systems, low-precision methods, and deployment-oriented optimization. Lite3R focuses on adapting strong geometry-grounded transformer backbones while jointly addressing structural and numerical efficiency.

  • Transformer-based 3D reconstruction: Transformer backbones improve global reasoning and cross-view correspondence but raise geometry-inference costs relative to earlier local or convolution-dominated pipelines.
  • Transformer-based 3D reconstruction: Recent systems redesign attention or introduce feed-forward sparse reconstruction variants to improve geometry-transformer efficiency.
  • Efficient attention for long-context geometry reasoning: Sparse, linear, and hybrid attention methods reduce complexity, but multi-view geometry also requires preserving interactions that carry cross-view correspondence cues.
  • Low-precision adaptation of pretrained geometry models: Geometry-sensitive models can suffer depth, pose, and 3D-consistency degradation from numerical errors, motivating joint structural lightweighting and low-precision adaptation.
  • System-oriented efficiency for end-to-end deployment: End-to-end deployment gains depend on memory traffic, activation storage, and scheduling in addition to kernel-level acceleration.

3 Method

Lite3R builds an efficient student from a dense pretrained 3D reconstruction teacher by combining Sparse Linear Attention, parameter-efficient FP8-aware training, and partial attention distillation. The framework preserves geometric representations while reducing token-mixing and deployment costs.

  • Dense teacher and lite student construction: The student copies the dense teacher, replaces attention modules with SLA blocks, and preserves geometry-critical components such as normalization, positional encoding, and task heads.The overall architecture remains largely intact while attention is structurally lightweighted.
  • Sparse Linear Attention: Lite3R replaces dense attention with Sparse Linear Attention, combining sparse geometric correspondences with low-cost global context.The sparse branch preserves high-value cross-view interactions, while the linear branch reduces token-mixing cost.
  • FP8-aware quantization-aware training: FP8-aware QAT exposes the student to low-precision perturbations during optimization to stabilize depth, pose, and 3D consistency after attention replacement.The method uses FP8 E4M3 fake quantization in the training computation path, while geometry-sensitive operators can remain in higher precision.
  • Parameter-efficient adaptation: Only lightweight SLA linear-branch projections are updated, while inherited backbone parameters remain frozen, reducing optimization cost and limiting drift from the teacher feature space.All student linear layers still undergo FP8 fake quantization in the forward pass, but gradients update only the lightweight projection layers.
  • Partial attention distillation and task supervision: Partial attention distillation aligns intermediate student and teacher attention outputs alongside the original geometry task objective.This guides the SLA-based, quantized student toward the teacher’s geometry-aware internal responses after structural and numerical changes.
  • Deployment pathway: Lite3R unifies structural lightweighting, FP8-aware QAT, partial attention distillation, and an FP8-compatible deployment pathway in a model-agnostic framework.The resulting system targets reduced inference and memory cost while preserving the geometric strengths of modern 3D backbones.

4 Experiments

Experiments evaluate Lite3R across VGGT and DA3-Large on BlendedMVS and DTU64, measuring reconstruction quality, deployment efficiency, component contributions, distillation strength, and training stability. Lite3R consistently reduces latency and memory while retaining competitive geometry, with SLA and FP8-aware QAT contributing different parts of the quality–efficiency tradeoff.

  • Experimental setup: Experiments use VGGT and DA3-Large on BlendedMVS and DTU64 under a controlled single-A100 evaluation setting.BlendedMVS reports depth, pose, point-cloud geometry, and efficiency; DTU64 reports pose errors and efficiency.
  • Main results: Lite3R reduces latency and memory across all four backbone–dataset pairs while keeping downstream geometry metrics within an acceptable range.The deployed configuration combines FP8-aware QAT with FP8 weight-only inference; native dynamic FP8 activation inference is not used on A100.
  • Main results: 1.76× speedup and 2.32× memory saving are achieved by VGGT-based Lite3R on BlendedMVS, while δ1 changes from 0.9930 to 0.9922.Chamfer distance improves from 0.2411 to 0.2354 and F5cm from 0.2005 to 0.2029.
  • Main results: 1.97× speedup and 1.98× memory saving are achieved by DA3-Large-based Lite3R on BlendedMVS, while AbsRel changes from 0.0862 to 0.0889.δ1 remains close to baseline, changing from 0.9329 to 0.9308, while F5cm improves from 0.1149 to 0.1210.
  • Ablation study: Removing FP8-aware QAT slightly improves quality but increases latency and memory from 274.38ms/2455MB to 377.21ms/4196MB, whereas removing SLA reduces speedup and memory saving to 1.21× and 1.86×.The ablation is restricted to VGGT on BlendedMVS; SLA mainly drives latency reduction and FP8-aware QAT mainly drives memory efficiency.
  • Ablation study: γ = 0.1 provides the most balanced distillation setting, while a 20-epoch FP8-aware QAT schedule produces geometric drift and the main results therefore use one epoch.All tested distillation settings have nearly identical efficiency, at about 1.75×–1.77× speedup and 2.32× memory saving.

5 Conclusion

Lite3R combines Sparse Linear Attention, parameter-efficient FP8-aware QAT, and partial attention distillation for efficient feed-forward 3D reconstruction. Across VGGT and DA3-Large on BlendedMVS and DTU64, it reduces latency and memory while maintaining competitive reconstruction quality.

  • Lite3R combines Sparse Linear Attention, parameter-efficient FP8-aware QAT, and partial attention distillation in a model-agnostic framework.The framework replaces dense attention and adapts only lightweight linear-branch projection layers.
  • 1.76–1.97× lower latency and 2.32–2.71× lower memory footprint are achieved across VGGT and DA3-Large experiments.The evaluation covers BlendedMVS and DTU64 while maintaining competitive depth, pose, and 3D reconstruction quality overall.
  • Both SLA and FP8-aware QAT are important for achieving the best quality–efficiency tradeoff.

A Sparse Linear Attention (SLA) summary

Sparse Linear Attention constructs the Lite3R student by combining sparse geometric interactions with a low-cost linear-context branch. This design preserves selected cross-view correspondences while reducing token-mixing cost.

  • SLA replaces dense self-attention with a hybrid sparse geometric branch and low-cost linear-context branch.The sparse branch retains high-value query–key interactions, while the linear branch provides global context.
  • SLA takes input tokens X ∈ R^N×d and produces output tokens O ∈ R^N×d using projected Q, K, and V features.The procedure uses frozen projection matrices WQ, WK, and WV, with a sparse keep ratio λ and trainable WO.
  • The sparse branch preserves selected query–key interactions that carry cross-view geometric correspondences.
  • The linear branch summarizes key–value information to provide low-cost global context.Its output is combined with the sparse branch through a trainable output projection.
  • The algorithm forms a TopK mask, computes sparse attention, constructs linear key–value and normalization summaries, and combines both branches.

B FP8-Aware Quantization-Aware Training

The FP8-aware QAT approach simulates low-precision effects during training while retaining full-precision gradient flow. It uses FP8 E4M3 quantization with separate scaling strategies for weights and activations.

  • FP8-aware QAT enables efficient deployment of large-scale transformer models while maintaining reconstruction quality.The approach is model-agnostic and can integrate with transformer-based architectures.
  • FP8 E4M3 uses 1 sign bit, 4 exponent bits, and 3 mantissa bits, with an approximately [−448, 448] dynamic range.Its reduced precision supports efficient inference on accelerators with native FP8 support.
  • FP8 fake-quantization simulates quantization noise during training while preserving full-precision gradients through a straight-through estimator.
  • Weight quantization uses per-output-channel scaling, whereas activation quantization uses per-token dynamic scaling.These strategies adapt to the local magnitude distributions of weights and activations.
  • The fake-quantization linear layer wraps a standard linear layer and can replace nn.Linear modules in transformer architectures.

B.5 Training Procedure

The training procedure first establishes a higher-precision baseline, then fine-tunes FP8 fake-quantized layers for a small number of epochs. Straight-through estimation preserves gradient computation while the model adapts to quantization noise.

  • The procedure begins by training or fine-tuning the original-precision model until convergence.This establishes a task-specific baseline in FP32, FP16, or BF16.
  • FP8 QAT fine-tuning replaces target linear layers with FP8 fake-quantization layers and continues training for typically 1–5 epochs.
  • During the forward pass, weights and activations are quantized to FP8 E4M3, while the backward pass uses the straight-through estimator.The learning rate is typically reduced by 10×, and trainable parameters may be all parameters or a selected subset.
  • The two-stage procedure adapts the model to quantization noise while leveraging representations learned during higher-precision training.
  • The FP8 fake-quantization linear layer accepts x ∈ R^N×d_in, W ∈ R^d_out×d_in, and b ∈ R^d_out, producing y ∈ R^N×d_out.It supports optional activation quantization and applies per-token activation and per-output-channel weight scaling.

B.6 Implementation Details

The framework supports selective quantization policies and numerical safeguards for FP8 fake-quantization, while keeping training overhead low.

  • Quantization policy: Selective quantization can target all linear layers, only attention and MLP layers, or exclude small projection heads.Normalization and embedding layers can remain in full precision, and layers with fewer than 1M parameters can be skipped.
  • Numerical stability: A small epsilon of 10^-8 prevents division by zero, while clamping keeps scaled values within the valid FP8 range.These safeguards address near-zero tensors and out-of-range values before casting.
  • Training efficiency: FP8 fake-quantization adds less than 5% training slowdown through efficient CUDA kernels and standard autograd backward computation.The straight-through estimator requires no additional backward computation beyond the standard autograd graph.

B.7 Deployment Considerations

Deployment stores scaling factors with FP8-quantized weights for accelerator inference, reducing memory and potentially increasing throughput, with slower simulated arithmetic on unsupported hardware.

  • Memory: FP8 weights occupy half the memory of FP16 weights and one-quarter the memory of FP32 weights.The reduction can support larger batch sizes or longer sequences.
  • Accuracy: FP8 QAT maintains task performance within 1-2% of higher-precision baselines across various vision-language tasks.This result is reported for the broader FP8 QAT setting rather than specifically for every Lite3R reconstruction experiment.
  • Hardware efficiency: Up to 2× throughput over FP16 is available from native FP8 tensor cores on supported hardware.The reported benefit applies to hardware with native FP8 support.
  • Evaluation overview: Figure 6 compares quality and efficiency metrics across nine settings, including backbones, datasets, and evaluation dimensions.The visualization compares Lite3R with corresponding higher-precision baselines.
  • Compatibility: Without native FP8 support, simulated FP8 arithmetic retains memory advantages but sacrifices speed benefits.The model is dequantized for computation and quantized again afterward.

C Supplementary Efficiency and Sensitivity Analysis

Supplementary analysis examines Lite3R’s global efficiency trends, DA3-Large’s parameter allocation, and layer-wise indicators of sensitivity to FP8 perturbation.

  • Global evaluation: Figure 6 provides a global comparison of Lite3R across backbones, datasets, and metric groups beyond the compact main-paper tables.The supplementary visualization broadens the view of reported efficiency and quality trends.
  • Efficiency analysis: DA3-Large trains only 0.11M of 411.06M parameters, or 0.03%, while freezing 99.97% of the model.Only projection layers corresponding to 28 replaced attention modules are updated, concentrating adaptation capacity in a tiny parameter subset.
  • Efficiency analysis: DA3-Large’s 0.03% trainable ratio is lower than VGGT’s 3.1%, providing less flexibility under structural and numerical changes.The adaptation remains concentrated in lightweight projection layers, alongside latency and memory benefits.
  • Sensitivity analysis: The sensitivity score combines dynamic range, the fraction of weights beyond 3σ, and kurtosis to characterize vulnerability to FP8 perturbation.The supplementary equation assigns weighted contributions to these statistical indicators.
  • Sensitivity analysis: Larger dynamic range, more outliers, and heavier tails indicate higher sensitivity to FP8 quantization.The indicators are computed for each linear layer’s weight tensor W.
Loading 2605.11354v1…