PaperlayerView exact revision on arXiv

Source-linked AI summary

MVSFormer: Multi-View Stereo by Learning Robust Image Features and Temperature-based Depth

Chenjie Cao, Xinlin Ren, Yanwei Fu

arXiv:2208.02541v3cs.CV

TL;DR

MVS feature learning remains challenged by reflections and texture-less regions, including when FPNs use pretrained CNNs. MVSFormer enhances MVS features with pretrained ViTs and combines classification and regression through temperature-based depth prediction. It achieves state-of-the-art DTU results and Top-1 rankings on both Tanks-and-Temples sets.

  • Problem

    FPNs and pretrained CNN features remain limited on reflections and texture-less regions, motivating exploration of pretrained ViTs for MVS feature learning.

    Source: §1 · p. 2

  • Method

    MVSFormer enhances MVS features with pretrained ViTs, including a frozen-DINO variant, and uses temperature-based depth prediction to unify classification and regression.

    Source: §3.1 · p. 6, §5 · p. 13

  • Results

    MVSFormer achieves state-of-the-art results on DTU and ranks Top-1 on both intermediate and advanced Tanks-and-Temples sets.

    Source: §4.1 · p. 10, §5 · p. 13

  • Takeaways & Limitations

    The method produces smooth depth maps and clear point clouds without outliers, while classification confidence can filter outliers.

    Source: §4.1 · p. 10, §5 · p. 13

    The temperature setting is fixed across datasets and instances and may not be optimal, although the reported difference is not obvious.

    Source: §4.1 · p. 9

Abstract

from arXiv · show

Feature representation learning is the key recipe for learning-based Multi-View Stereo (MVS). As the common feature extractor of learning-based MVS, vanilla Feature Pyramid Networks (FPNs) suffer from discouraged feature representations for reflection and texture-less areas, which limits the generalization of MVS. Even FPNs worked with pre-trained Convolutional Neural Networks (CNNs) fail to tackle these issues. On the other hand, Vision Transformers (ViTs) have achieved prominent success in many 2D vision tasks. Thus we ask whether ViTs can facilitate feature learning in MVS? In this paper, we propose a pre-trained ViT enhanced MVS network called MVSFormer, which can learn more reliable feature representations benefited by informative priors from ViT. The finetuned MVSFormer with hierarchical ViTs of efficient attention mechanisms can achieve prominent improvement based on FPNs. Besides, the alternative MVSFormer with frozen ViT weights is further proposed. This largely alleviates the training cost with competitive performance strengthened by the attention map from the self-distillation pre-training. MVSFormer can be generalized to various input resolutions with efficient multi-scale training strengthened by gradient accumulation. Moreover, we discuss the merits and drawbacks of classification and regression-based MVS methods, and further propose to unify them with a temperature-based strategy. MVSFormer achieves state-of-the-art performance on the DTU dataset. Particularly, MVSFormer ranks as Top-1 on both intermediate and advanced sets of the highly competitive Tanks-and-Temples leaderboard.

1 Instruction

Learning reliable features is central to generalizing MVS, but FPNs and pretrained CNN features remain weak on reflections and texture-less regions. MVSFormer uses pretrained ViTs to enhance feature learning, with multi-scale training and temperature-based depth prediction addressing resolution generalization and the trade-offs between classification and regression.

1 Instruction

  • FPN features and pretrained CNNs remain limited on reflections and texture-less regions, while cost-volume regularization cannot fully resolve ambiguous feature matches.CNN low-level features have limited receptive fields, and high-level features are too semantically abstract for fine-grained matching.Source: §1 · p. 2
  • Pretrained ViTs enhance FPN features in MVSFormer, with hierarchical Twins and efficient attention supporting high-resolution training and better results than FPNs.A plain-ViT alternative, MVSFormer-P, freezes a DINO backbone to reduce computation while retaining competitive performance.Source: §1 · pp. 2–3
  • Multi-scale training generalizes MVSFormer to test resolutions of 1536 or 1920, above the largest training resolution of 1280.Gradient accumulation is used to improve GPU utilization during multi-scale training.Source: §1 · p. 3
  • Cross-entropy training yields more reliable confidence maps but slightly worse depth predictions because argmax cannot provide exact depth.MVSFormer instead predicts temperature-based depth expectation at inference, producing smooth depth predictions and superior final point clouds.Source: §1 · p. 3
  • Pretrained Twins-small MVSFormer reduces DTU point-cloud overall error from 0.312 to 0.289 versus pretrained ResNet, with other model settings unchanged.The paper also reports state-of-the-art results on DTU and Top-1 rankings on both Tanks-and-Temples sets.Source: §1 · p. 3

2 Related Works

Prior MVS work has improved cost volumes, FPN features, and feature dependencies, but systematic use of pretrained ViTs for MVS feature learning was lacking. The paper situates its approach among these lines of work and discusses the resolution and data demands of ViTs.

2 Related Works

  • MVS methods have improved cost-volume construction and regularization, while this paper explores pretrained ViTs for feature learning as an orthogonal direction.Prior approaches include coarse-to-fine models, RNNs, visibility or epipolar aggregation, and hybrid 3D U-Net regularization.Source: §2 · p. 3
  • FPN feature-learning work uses deformable convolutions, curvature-guided kernels, and attention, but retains CNN inductive biases and lacks systematic pretrained-ViT exploration for MVS.The cited works include, (Mi et al., 2021; Ding et al., 2021), and (Xu et al., 2020; Giang et al., 2021).Source: §2 · p. 4
  • ViTs offer long-range modeling and multiple pretraining approaches, but their weaker CNN-like inductive biases make generalization more data-hungry.The passage distinguishes supervised classification pretraining from self-supervised masked-prediction and contrastive-learning approaches.Source: §2 · p. 4
  • Multi-scale training is used in ViTs for high-resolution tasks, and the paper evaluates it for MVS while repurposing gradient accumulation for GPU utilization.The paper identifies multi-scale training as key to releasing ViTs’ potential for MVS.Source: §2 · p. 4

3 Method

MVSFormer combines pretrained ViT features with FPN features for multi-view depth estimation, then uses multi-stage cost volumes to predict depth. Its method also includes multi-scale training and a temperature-based inference strategy that combines classification- and regression-based depth prediction.

3.1 Feature Extraction

  • MVSFormer adds half-resolution pretrained ViT features to FPN features, using trainable hierarchical Twins or frozen DINO attention maps with a trainable GLU in MVSFormer-P.ViT features provide global correlations while FPN features learn detailed ones; the GLU uses DINO attention maps to protect important features during dimension reduction.Source: §3.1 · pp. 5–6

3.2 Efficient Multi-scale Training

  • Dynamic training resolutions from 512 to 1280, paired with gradient accumulation and resolution-dependent sub-batch sizes, generalize MVSFormer to at least 2K Tanks-and-Temples images.The strategy randomly samples aspect ratios from 0.8 to 0.67 and keeps the largest batch size by splitting batches into sub-batches for gradient accumulation.Source: §3.2 · p. 7

3.3 Correlation Volume Construction

  • For each depth hypothesis, source features are warped and grouped correlations are fused across views using learned visibility weights before 3D U-Net cost-volume regularization.The regularized cost volume produces pixel-wise depth-hypothesis probabilities at each stage.Source: §3.3 · p. 7

3.4 Temperature-based Depth Prediction

  • Classification confidence can retain the DTU kettle handle, while regression can remain highly confident in out-of-range depths and struggle to filter outliers without harming correct points.The authors train with masked depth outliers because cross-entropy cannot handle out-of-range labels; they report masked L1 regression performs worse than regular regression.Source: Figure 4 · p. 8, §3.4 · p. 8
  • At inference, temperature-scaled softmax interpolates between argmax classification at t = ∞ and regression at t = 1, using higher temperatures early and lower temperatures later.The reported stage temperatures are {5, 2.5, 1.5, 1}; temperature adjustment is used only at test time, while training uses classification depth with cross-entropy.Source: §3.4 · pp. 8–9

4 Experiments

Experiments evaluate MVSFormer on DTU, Tanks-and-Temples, and ETH3D, then ablate its pretrained backbones, multi-scale training, and depth-prediction choices. The full model leads key benchmark results, while the ablations examine how pretraining, scale variation, attention maps, and temperature affect performance.

4 Experiments

  • Evaluation covers DTU, Tanks-and-Temples, and ETH3D; models trained on DTU are finetuned on BlendedMVS for generalization to more complex environments.DTU evaluation uses five views and a fixed 1152 × 1536 resolution; the ETH3D evaluation uses the model without retraining.Source: §4 · p. 9, §4.1 · pp. 9–10

4.1 Quantitative Results

  • 66.37 and 40.87 mean F-scores make MVSFormer best on Tanks-and-Temples intermediate and advanced sets; it also leads ETH3D F1 and DTU completeness and overall error.The Tanks-and-Temples submission ranked Top-1 on both sets; the full trainable model achieved the best completeness and overall error on DTU.Source: Table 2 · p. 10, Table 3 · p. 10, §4.1 · p. 10

4.2 Ablation studies

  • DINO-small outperforms MAE-base among frozen ViTs with fewer parameters, while Twins and DINO MVSFormers respectively achieve the best point-cloud and depth results.ResNet50 improves depth but does not reduce point-cloud overall error; MVSFormers improve the baseline even without multi-scale training.Source: §4.2 · p. 11, Table 4 · p. 11
  • Multi-scale training improves both Twins- and DINO-based MVSFormers, with a 0.19 overall-error gain for trainable Twins; fixed-resolution finetuning performs worse.The multi-scale strategy also outperforms finetuning for five epochs on fixed 1024×1280 images.Source: §4.2 · p. 11, Table 4 · p. 11
  • Twins-small without pretraining underperforms its pretrained version, indicating that pretraining is critical to the proposed MVSFormer.The unpretrained version performs similarly to TransMVSNet, except for the temperature-based depth prediction.Source: §4.2 · p. 12, Table 5 · p. 12
  • Both attention-map fusion strategies improve MVSFormer-P over the baseline, and GLU yields better depth than concatenation at equal computation.The ablation compares fusion of the DINO [CLS] attention map using GLU against simple concatenation with parameter-balancing convolutions.Source: §4.2 · p. 12, Table 5 · p. 12
  • CLA produces more reliable confidence maps than REG for separating certain foregrounds from uncertain backgrounds, yielding more valid points and fewer outliers.The comparison is illustrated on real-world Tanks-and-Temples reconstructions.Source: §4.2 · p. 12, Figure 6 · p. 12
  • As temperature falls, depth smooths and point clouds improve, but t = 0.75 worsens overall error; staged {5,2.5,1.5,1} balances metrics better than any constant t.The schedule may not be optimal, but it is fixed across datasets and instances without data dependency.Source: §4.1 · p. 9, §4.2 · pp. 12–13

5 Conclusion

MVSFormer combines pre-trained hierarchical ViTs with FPNs, multi-scale training, and temperature-based depth prediction. The paper reports state-of-the-art DTU results and Top-1 rankings on both Tanks-and-Temples sets.

5 Conclusion

  • MVSFormer achieves state-of-the-art results on DTU and ranks Top-1 on both intermediate and advanced Tanks-and-Temples sets.Source: §5 · p. 13
  • Pre-trained hierarchical ViT Twins improve MVSFormer’s feature encoding, while frozen-backbone MVSFormer-P achieves competitive results.Source: §5 · p. 13
  • Temperature-based depth prediction unifies REG and CLA, producing smooth depth maps and clear point clouds without outliers.Source: §5 · p. 13

A.1 Multi-scale Training

The appendix documents MVSFormer’s training, inference, and design choices, including component comparisons and scope limits for confidence estimates.

A.1 Multi-scale Training

  • Training uses 25 random scale patterns, 0.83–1.0 image crops, and a global batch size of 8 across two 32GB GPUs.Scale heights range from 512 to 1024 and widths from 640 to 1280; 10-epoch training takes about 22 hours on DTU and 15 hours on BlendedMVS.Source: §A.1 · p. 18, Table 7 · p. 18

A.2 Data Augmentation

  • Shared random gamma and color-jitter parameters for reference and source views slightly improve depth performance.The listed augmentation ranges cover gamma, brightness, contrast, saturation, and hue.Source: §A.2 · p. 18, Table 8 · p. 19

A.3 Post-processing

  • DTU point clouds use Gipuma fusion with fixed disparity, consistency, and probability thresholds; Tanks-and-Temples uses dynamic consistency checking.Source: §A.3 · p. 19

A.4 Confidence Maps

  • CLA confidence is the maximum depth-hypothesis probability, while REG confidence averages probabilities along depth before gathering confidence.Confidence maps from all stages are averaged after resizing to the original image size; the pooling kernels vary across the four stages.Source: §A.4 · p. 19

A.5 Additional Discussions

  • CLA and REG confidence represent training-distribution noise, not out-of-data uncertainty; wrong preset depth ranges can yield improper results.The discussion also attributes FPN’s continuing role to full-resolution detail alongside ViT features, and prefers [CLS] attention for its global semantics and smaller map.Source: §A.5 · pp. 19–20

B Inference Memory and Time Costs

  • MVSFormer adds parameters but costs only a little more GPU memory and time than the baseline; Twins-based MVSFormer infers faster than MVSFormer-P.At 1152 × 1536, the comparison uses a V100 GPU; CDS-MVSNet is slightly more efficient but is discouraged in performance.Source: B Inference Memory and Time Costs · p. 20, Table 9 · p. 20

C.1 Pre-trained ViTs without FPNs

  • ViTs lack CNN-level detail at 2mm and 4mm depth errors but are more robust at 8mm and 14mm; Twins-small beats FPN except at 2mm.These comparisons use low-resolution DTU cases; the table reports depth metrics against FPN and pre-trained CNNs and ViTs.Source: §C.1 · p. 20, Table 10 · p. 21

C.2 Different Feature Fusion Strategies of MVSFormer

  • Multi-scale Feature Addition slightly improves depth predictions but worsens point-cloud metrics, so the paper adopts Direct Feature Addition.The comparison adds upsampled ViT features at 1/4 and 1/2 scales; ViT features are considered unsuitable for high-resolution features.Source: §C.2 · p. 21, Table 11 · p. 21

C.3 Qualitative Results of Temperature-based Depth Prediction

The appendix compares temperature settings and ViT backbone capacities, and reports a ResNet50 experiment using the other proposed components.

C.3 Qualitative Results of Temperature-based Depth Prediction

  • The {5, 2.5, 1.5, 1} temperature setting yields less surface error below 2mm than t = 100 or t = ∞, with fewer side effects than t = 1.The authors describe early-stage CLA followed by later-stage REG as a reasonable balance for depth detail and point-cloud results.Source: Figure 7 · p. 21, §C.3 · p. 21

C.4 Comparison of Different ViT Capacities

  • DINO-base performs worse than DINO-small, while Twins-base improves depth metrics over Twins-small but yields negligible point-cloud improvements.Both base models converge faster than their small counterparts.Source: §C.4 · p. 22, Table 12 · p. 22

C.5 The Performance of Pre-trained ResNet50 with All Other Techniques

  • Multi-scale training does not improve pre-trained ResNet50 substantially, consistent with CNNs’ spatial invariance making it less important for ResNet50.Source: §C.5 · p. 22, Table 13 · p. 22

C.6 More Detailed Quantitative Analysis of REG, CLA, and Temperature-based CLA

The analysis compares REG, CLA, and temperature-based CLA on point-cloud metrics, then examines alternative temperature schedules and their trade-offs.

C.6 More Detailed Quantitative Analysis of REG, CLA, and Temperature-based CLA

  • Temperature-based CLA achieves better Acc and Ovl than REG with similar Cop; vanilla CLA has worse Acc (0.349 vs 0.336) despite similar Cop (0.248 vs 0.249).The authors attribute the temperature-based result to better confidence maps filtering boundary outliers and to resolving inexact argmax depth predictions.Source: §C.6 · p. 22, Table 14 · p. 23
  • The {∞, ∞, ∞, 1} schedule has the best depth metric and overall point-cloud reconstruction, but a larger accuracy–completeness gap than {5, 2.5, 1.5, 1}.The {5, 5, 5, 1} schedule outperforms {5, 2.5, 1.5, 1} on most metrics except completeness, with an unimpressive improvement; lowering temperature at the last stage reduces 1mm and 2mm errors at negligible disparity cost.Source: §C.6 · pp. 22–23, Table 15 · p. 23

C.7 Ablations about More Source Views

Adding more source views improves MVSFormer’s Tanks-and-Temples results in complex scenes. Qualitative results also show robust depth, confidence, and reconstruction quality on DTU and Tanks-and-Temples.

C.7 Ablations about More Source Views

  • 20-view inputs produce more reliable depth maps than 10-view inputs, and adding source views further improves MVSFormer’s Tanks-and-Temples results.The authors report that the improvement extends to complex scenes and that their state-of-the-art results can be further improved with more source views.Source: §C.7 · p. 23, Figure 8 · p. 23, Table 16 · p. 24

C.8 Qualitative Results of DTU

  • On DTU, MVSFormer produces robust, smooth depth maps and reliable confidence maps, while reconstructing challenging point clouds competitors omit.The passage contrasts its depth with CDS-MVSNet and notes instability in GBiNet’s argmax predictions.Source: §C.8 · p. 24, Figure 9 · p. 24

C.9 Qualitative Results of Tanks-and-Temples

  • On Tanks-and-Temples, MVSFormer’s depth and confidence maps yield high-quality filtered depth at a confidence threshold above 0.5.The qualitative example is shown in Figure 11.Source: §C.9 · p. 24
  • On qualitative Tanks-and-Temples intermediate-set Horse and Lighthouse results, MVSFormer reconstructs more detail (better Recall) and produces more accurately positioned point clouds (better Precision) than TransMVSNet and UniMVSNet.Source: §C.9 · p. 24

C.10 Training Randomness of MVSFormer

Additional runs with different random seeds test the stability of MVSFormer and MVSFormer-P. The reported reconstruction results are stable.

C.10 Training Randomness of MVSFormer

  • Reconstruction results are stable across additional MVSFormer and MVSFormer-P runs using different random seeds.The additional Twins-based MVSFormers and DINO-based MVSFormer-P models use the same settings except for random seeds.Source: §C.10 · p. 24

D Real-World Results

Real-world cases compare MVSFormer with CasMVSNet without masking. MVSFormer produces good results in these examples, including on texture-less and reflective objects.

D Real-World Results

  • Without segmentation masks, CasMVSNet produces messy outliers, while MVSFormer achieves good results, including on texture-less and reflective objects.The comparisons use Colmap camera poses and the same Gipuma fusion settings, with each method’s DTU confidence threshold.Source: D Real-World Results · pp. 25–26, Figure 13 · p. 26
  • The real-world examples include complex structures and hairy, texture-less, and reflective objects.Figure 13 compares point clouds from two cases with CasMVSNet.Source: D Real-World Results · p. 25, Figure 13 · p. 26

E More Point Cloud Results

The paper presents additional MVSFormer point clouds from DTU, Tanks-and-Temples, and ETH3D.

E More Point Cloud Results

F Limitations and Future Works

The authors identify limits in comparing pre-training strategies and in the fusion methods explored, and flag potential societal impacts from real-world use of 3D reconstruction.

F Limitations and Future Works

  • Pre-training comparisons are confounded by differences in ViT architecture and training settings; matched architectures and tasks are left for future work.The authors also note that re-training these models would demand extraordinary computing resources.Source: F Limitations and Future Works · p. 27
  • The study uses only single- and multi-scale feature addition for fusion, leaving cross-attention and GRU modules as future directions.The authors say their simple fusion is sufficient for competitive performance.Source: F Limitations and Future Works · p. 27

G Broader Impact

  • Real-world use of 2D-image-based 3D reconstruction may involve controversial images, so potential negative societal impacts require further consideration.Source: G Broader Impact · p. 27

Related papers

Loading 2208.02541v3…