PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantized Visual Geometry Grounded Transformer

Weilun Feng, Haotong Qin, Mingqiang Wu, Chuanguang Yang, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Yulun Zhang, Michele Magno, Yongjun Xu

arXiv:2509.21302v4cs.CV

TL;DR

Billion-scale VGGTs are costly to deploy, and their special tokens and multi-view inputs create challenges for quantization and calibration. QuantVGGT combines smoothing-based quantization with noise-filtered, frame-aware calibration sampling. It achieves state-of-the-art performance across bit-widths, including an AUC@30 of 88.2 under W4A4 compared with QuaRot’s 81.6 under 20 frames.

  • Problem

    Billion-scale VGGTs incur prohibitive computational and memory costs, while their data-independent tokens and complex multi-view inputs make quantization and calibration challenging.

    Source: §1 · p. 2

  • Method

    QuantVGGT combines global Hadamard rotation and local channel smoothing with outlier filtering and frame-aware diverse calibration sampling.

    Source: §1 · p. 2

  • Results

    QuantVGGT achieves state-of-the-art performance across bit-widths; under W4A4, it reaches 88.2 AUC@30 versus QuaRot’s 81.6 AUC@30 under 20 frames.

    Source: §4.2 · p. 9, §5 · p. 10

  • Takeaways & Limitations

    The reported low-bit results indicate that QuantVGGT preserves reconstruction performance while improving memory and inference efficiency.

    Source: §1 · p. 3

    The calibration sampling principle assumes the input space can be divided into domains with finite, disjoint sub-regions.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Learning-based 3D reconstruction models, represented by Visual Geometry Grounded Transformers (VGGTs), have made remarkable progress with the use of large-scale transformers. Their prohibitive computational and memory costs severely hinder real-world deployment. Post-Training Quantization (PTQ) has become a common practice for compressing and accelerating models. However, we empirically observe that PTQ faces unique obstacles when compressing billion-scale VGGTs: the data-independent special tokens induce heavy-tailed activation distributions, while the multi-view nature of 3D data makes calibration sample selection highly unstable. This paper proposes the first Quantization framework for VGGTs, namely QuantVGGT. This mainly relies on two technical contributions: First, we introduce Dual-Smoothed Fine-Grained Quantization, which integrates pre-global Hadamard rotation and post-local channel smoothing to mitigate heavy-tailed distributions and inter-channel variance robustly. Second, we design Noise-Filtered Diverse Sampling, which filters outliers via deep-layer statistics and constructs frame-aware diverse calibration clusters to ensure stable quantization ranges. Comprehensive experiments demonstrate that QuantVGGT achieves the state-of-the-art results across different benchmarks and bit-width, surpassing the previous state-of-the-art generic quantization method with a great margin. We highlight that our 4-bit QuantVGGT can deliver a 3.7$\times$ memory reduction and 2.5$\times$ acceleration in real-hardware inference, while maintaining reconstruction accuracy above 98\% of its full-precision counterpart. This demonstrates the vast advantages and practicality of QuantVGGT in resource-constrained scenarios. Our code is released in https://github.com/wlfeng0509/QuantVGGT.

1 INTRODUCTION

Billion-scale VGGTs are costly to deploy, and their data-independent tokens and complex multi-view inputs make quantization and calibration challenging. QuantVGGT addresses these challenges with two tailored techniques and reports effective low-bit quantization with efficiency gains while preserving reconstruction accuracy.

1 INTRODUCTION

  • VGGT’s data-independent camera and register tokens create heavy-tailed activations, while complex multi-view data makes representative calibration sampling unstable.The paper identifies both as model-specific quantization challenges.Source: §1 · p. 2
  • Dual-Smoothed Fine-Grained Quantization combines pre-global Hadamard rotation with post-local channel smoothing to mitigate heavy tails and channel-level variance.The paper also characterizes this dual-stage scheme as reducing quantization errors.Source: §1 · p. 2
  • Noise-Filtered Diverse Sampling filters outliers using deep-layer activation statistics and builds frame-aware clusters for representative, stable calibration.The strategy uses VGGT’s inductive bias when constructing the clusters.Source: §1 · p. 2
  • Extensive experiments show that QuantVGGT enables effective low-bit VGGT quantization, with memory and inference efficiency gains while preserving reconstruction accuracy.The introduction reports these outcomes without specifying their values.Source: §1 · p. 3

2 RELATED WORKS

Learning-based 3D reconstruction models such as VGGT predict multiple 3D attributes from images, but their scale and computational demands limit deployment. Existing PTQ methods perform well on language and 2D vision models, while VGGT-specific quantization remains underexplored.

2.1 LEARNING-BASED 3D RECONSTRUCTION

  • VGGT predicts camera position, dense depth, point maps, and point tracking in one forward process, but its 1.2B parameters and computational complexity limit deployment.Earlier learning-based methods include DUSt3R and MASt3R, which respectively predict scene point maps and refine metric-scale approximation.Source: §2.1 · p. 3

2.2 MODEL QUANTIZATION

  • PTQ methods address issues such as imbalanced distributions, but methods effective on language and 2D vision models do not generalize well to large-scale VGGT.The paper presents its approach as the first PTQ framework specially designed for VGGT and intended to retain performance at low bit-widths.Source: §2.2 · pp. 3–4

3 METHODS

VGGT processes image sequences with frame-specific camera and register tokens, while PTQ compresses its weights and activations using a small calibration set. QuantVGGT addresses token-induced skew and calibration instability with dual-smoothed quantization and noise-filtered, frame-aware sampling.

3.1.1 VISUAL GEOMETRY GROUNDED TRANSFORMER

  • VGGT tokenizes each frame and appends one camera token and four register tokens, using a first-frame set and a shared set for later frames.These special tokens aggregate different 3D attributes, such as camera parameters and scene geometry.Source: §3.1.1 · p. 4

3.1.2 POST-TRAINING QUANTIZATION

  • PTQ keeps full-precision weights fixed and tunes quantization parameters using a relatively small calibration dataset.Symmetric quantization maps floating-point values to low-bit integers, using rounding and clamping to the valid range.Source: §3.1.2 · p. 4

3.2 DUAL-SMOOTHED FINE-GRAINED QUANTIZATION

  • DSFQ applies Hadamard rotation to disperse special-token outliers, then channel-wise scaling in rotated space; weights are output-dimension-wise and activations token-wise quantized.The post-local scaling addresses residual local variance after rotation, and its scale factors can be fused into neighboring normalization layers without runtime cost.Source: §3.2 · pp. 5–6

3.3 NOISE-FILTERED DIVERSE SAMPLING

  • NFDS filters deep-layer outliers, then clusters samples by first-to-subsequent-frame feature similarity and uniformly samples each cluster for calibration.The frame-relative clustering reflects VGGT’s inductive bias; raw labels are suboptimal because geometry scenes can contain multiple objects and labels may not represent their semantics directly.Source: §3.3 · pp. 6–8

4 EXPERIMENTS

The experiments evaluate VGGT-1B on camera pose and point-map reconstruction across datasets and bit widths, then test QuantVGGT’s architecture, sampling strategy, and deployment efficiency. Results show strong low-bit performance, with ablations supporting the combined smoothing and sampling designs.

4.1 EXPERIMENTAL AND EVALUATION SETTINGS

  • Experiments use VGGT-1B for CO3Dv2 camera-pose estimation and DTU point-map estimation, with W8A8 and W4A4 quantization settings.Camera-pose evaluation uses 10 randomly sampled frames and a 20-frame generalized evaluation; baselines include PTQ, vision-transformer, and language-transformer quantizers.Source: §4.1 · p. 8, §4.2 · p. 8

4.2 MAIN RESULTS

  • At W4A4 under 20 frames, QuantVGGT reaches AUC@30 88.2 versus QuaRot’s 81.6, retaining 98% of full-precision performance.Under W8A8, QuantVGGT preserves 99.9% performance, with AUC@30 of 89.4 versus 89.5 for FP.Source: §4.2 · p. 9
  • On 7-Scenes and NRGBD, W4A4 QuantVGGT substantially outperforms SmoothQuant and QuaRot while maintaining nearly lossless performance.Table 3 reports point-cloud reconstruction experiments on VGGT across both datasets.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 ABLATION STUDY

  • DSFQ’s pre-global rotation followed by post-local smoothing performs better than smoothing before rotation, which weakens smoothing by rearranging channel outliers.Naive quantization collapses to AUC@3 of 9.7; rotation-only and scale-only methods improve distributions but still degrade, while DSFQ combines both with fine-grained quantization.Source: §4.3 · pp. 9–10, Table 4 · p. 10
  • NFDS combines outlier removal with frame-aware clustering to preserve average performance and improve stability across five random seeds.Random sampling has substantial variance; filtering reduces variance, while clustering improves diversity and mean performance but leaves outlier-related variance.Source: §4.3 · p. 10, Figure 5 · p. 10

4.4 EFFICIENCY ANALYSIS

  • At W4A4, DSFQ adds only 0.2% latency over naive quantization while preserving performance, and QuantVGGT exceeds naive W8A8 performance.Measurements use one NVIDIA RTX 4090 24G GPU with CUDA 12.4 and CUTLASS low-bit INT matrix multiplication.Source: §4.4 · p. 10

5 CONCLUSION

QuantVGGT is presented as the first PTQ framework for VGGTs, addressing token-driven heavy-tailed distributions and unstable multi-view calibration. Experiments report state-of-the-art performance across bit widths and benchmarks, surpassing existing quantization methods.

5 CONCLUSION

  • QuantVGGT is presented as the first PTQ framework for VGGTs, targeting quantization-unfriendly distributions from data-independent tokens and unstable multi-view calibration.The framework combines Dual-Smoothed Fine-Grained Quantization with Noise-Filtered Diverse Sampling.Source: §5 · p. 10
  • Dual-Smoothed Fine-Grained Quantization smooths heavy-tailed distributions, while Noise-Filtered Diverse Sampling constructs frame-aware diverse calibration clusters for stable datasets..Source: §5 · p. 10
  • Experiments report state-of-the-art performance across bit widths and benchmarks, substantially surpassing existing quantization methods.Figure 6 is captioned as reporting latency speedup and corresponding performance under 20 frames.Source: §5 · p. 10, Figure 6 · p. 10

7 ETHICS STATEMENT

The statement reports no ethics-related risks and says the work follows the ICLR Code of Ethics. It characterizes the work as algorithmic research on inference acceleration and compression using public open-source models.

7 ETHICS STATEMENT

  • The authors state that the research follows the ICLR Code of Ethics and presents no ethics-related risks.The statement rules out identified scenarios involving public safety, privacy infringement, or discrimination.Source: §7 · p. 11
  • The work uses public open-source VGGT models and focuses on algorithmic innovation for inference acceleration and compression.Source: §7 · p. 11
  • The ethics statement says the research does not involve scenarios endangering public safety, infringing privacy, or producing discrimination.Source: §7 · p. 11

8 REPRODUCIBILITY STATEMENT

The reproducibility statement points to detailed configurations, method descriptions, and evaluation metrics, and says comparative results follow baseline configurations. The theorem proof maximizes information by assigning equal probabilities across the finite support.

8 REPRODUCIBILITY STATEMENT

  • Experimental configurations, method details, and evaluation metrics are described in Section 4.1 and Appendix Section B, with comparisons following baseline configurations.The statement also notes supplementary raw reconstruction files and a public code release upon publication.Source: §8 · p. 11

A PROOF OF THEOREM 3.2

  • The proof concludes that assigning probability 1/N* to each supported outcome maximizes information.The argument states that the support size N* is finite.Source: A PROOF OF THEOREM 3.2 · p. 15
  • The proof reduces the information objective to maximizing Hi(D), then obtains equal probabilities across the finite support and maximum entropy.It uses sample independence to make Eq. 14 equivalent to the displayed objective.Source: A PROOF OF THEOREM 3.2 · p. 15

B EXPERIMENT SETTINGS

The experiments evaluate reconstruction and pose estimation under specified frame-selection protocols, using channel-wise weight quantization and token-wise activation quantization. Calibration uses a selected 40-sample subset and the reported settings define its scale and clustering parameters.

B EXPERIMENT SETTINGS

  • Camera pose is evaluated using 10- and 20-frame sequences with AUC, while point-map estimation samples keyframes every five images and reports Accuracy, Completion, and Normal Consistency.AUC combines Relative Rotation Accuracy and Relative Translation Accuracy.Source: B EXPERIMENT SETTINGS · p. 16
  • Weights use channel-wise symmetric quantization, activations use dynamic token-wise quantization, and calibration follows a block-wise pipeline.Hadamard rotation uses a random Hadamard matrix.Source: B EXPERIMENT SETTINGS · p. 16
  • Calibration selects 40 of 400 samples, with α = 0.5, p = 0.2, K = 8, and learnable channel scales and quantization parameters.The learning rates are 5e−3 for channel scale and 5e−2 for quantization parameters.Source: B EXPERIMENT SETTINGS · p. 16
  • The additional evaluation tables cover CO3Dv2 quantization settings, Fast3R point-cloud reconstruction on 7-Scenes, and acceleration-method comparisons on CO3Dv2.The supplied table captions specify scope but do not provide result values.Source: Table 5 · p. 16, Table 6 · p. 16, Table 7 · p. 16

C MORE EXPERIMENTS RESULTS

Additional experiments test QuantVGGT across quantization formats, a different reconstruction architecture, and an alternative acceleration method. The reported results describe strong accuracy, including practically lossless Fast3R performance and higher accuracy than FastVGGT across the compared metrics.

C MORE EXPERIMENTS RESULTS

  • Mixed-precision and asymmetric W4A4 settings maintain strong accuracy, while one-epoch QAT improves low-bit performance at substantial computational overhead.The lightweight QAT experiment fine-tunes on the same calibration dataset without extra data.Source: C MORE EXPERIMENTS RESULTS · pp. 16–17
  • QuantVGGT achieves practically lossless performance against full-precision Fast3R and substantially surpasses QuaRot and SmoothQuant on every reported 7-Scenes metric.The evaluation uses Fast3R for point-cloud reconstruction on 7-Scenes.Source: C MORE EXPERIMENTS RESULTS · p. 17
  • QuantVGGT achieves larger FLOPs reduction than FastVGGT (1.40T vs. 1.68T) while maintaining significantly higher accuracy at all metrics.The comparison is for camera pose estimation on CO3Dv2; the methods target different aspects and are described as conceptually orthogonal.Source: C MORE EXPERIMENTS RESULTS · p. 17

D MORE ANALYSIS ON DUAL-SMOOTHED FINE-GRAINED QUANTIZATION

The analysis examines token-induced activation saliency and the latency–performance trade-offs of DSFQ. It reports accuracy gains from fine-grained quantization and Hadamard rotation, while finding robustness to modest hyperparameter perturbations.

D MORE ANALYSIS ON DUAL-SMOOTHED FINE-GRAINED QUANTIZATION

  • Data-independent tokens show stronger salient activation patterns than adjacent image tokens across different VGGT layers, creating a quantization bottleneck.The passage attributes the amplified saliency to special registration tokens.Source: D MORE ANALYSIS ON DUAL-SMOOTHED… · p. 18
  • Dynamic token-wise quantization improves AUC@30 from 82.2 to 86.9 over static tensor-wise quantization with only 0.01% additional latency.The analysis reports that dynamic and token-wise quantization together add almost no memory burden.Source: D MORE ANALYSIS ON DUAL-SMOOTHED… · p. 18
  • Hadamard rotation improves AUC@30 from 81.9 to 86.9 with almost no additional burden when deployed using a fast CUDA Hadamard-transform kernel.The smooth factor is fused into the preceding LayerNorm operation.Source: D MORE ANALYSIS ON DUAL-SMOOTHED… · p. 18, Table 9 · p. 18
  • Ablations of the Hadamard matrix and smoothing factor indicate robustness to small perturbations without heavy hyperparameter tuning.The cited table covers Hadamard and smooth-factor sensitivity on CO3Dv2 under W4A4.Source: D MORE ANALYSIS ON DUAL-SMOOTHED… · p. 18, Table 10 · p. 18

E MORE ANALYSIS ON NOISE-FILTERED DIVERSE SAMPLING

The further analyses report NFDS robustness across settings, calibration and inference costs, and visual comparisons of quantized reconstruction. Frame-based clustering performs favorably in the reported strategy comparisons, while the additional computation analysis describes calibration overhead and acceleration scaling.

E MORE ANALYSIS ON NOISE-FILTERED DIVERSE SAMPLING

  • NFDS performance is robust across tested hyperparameters and steadily improves with more calibration samples.The ablation varies threshold p, cluster count K, and sample size N for W4A4 QuantVGGT on CO3Dv2 with 10 frames.Source: E MORE ANALYSIS ON NOISE-FILTERED… · p. 18, Table 11 · p. 19
  • Frame-based clustering yields better average performance and less randomness impact than random sampling; feature- and label-based clustering offer only slight improvements over random sampling.The comparison uses five random seeds.Source: E MORE ANALYSIS ON NOISE-FILTERED… · p. 19, Figure 8 · p. 19
  • On CO3Dv2 sequences whose first frame has average cosine similarity below 0.1 to the others, NFDS consistently outperforms random sampling with 20 calibration samples.The passage attributes this to filtering low-quality outliers and clustering relative inter-frame geometry rather than relying only on first-frame appearance.Source: E MORE ANALYSIS ON NOISE-FILTERED… · p. 19, Table 12 · p. 19

F CALIBRATION AND INFERENCE COMPUTATION RESOURCE

H MORE PERFORMANCE VISUALIZATION

Related papers

Loading 2509.21302v4…