PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-view 3D Reconstruction with Transformer

Dan Wang, Xinrui Cui, Xun Chen, Zhengxia Zou, Tianyang Shi, Septimiu Salcudean, Z. Jane Wang, Rabab Ward

arXiv:2103.12957v1cs.CV

TL;DR

CNN-based methods separate multi-view feature extraction and fusion, and often leave relationships among views unexplored. VolT reframes reconstruction as sequence-to-sequence prediction in a unified Transformer; on ShapeNet it reports state-of-the-art accuracy with 30% of the parameters used by other recent CNN-based methods.

  • Problem

    CNN-based methods usually investigate feature extraction and view fusion separately, rarely explore relationships among views, and may saturate as the number of input views increases.

    Source: §I · p. 1

  • Method

    VolT frames multi-view reconstruction as sequence-to-sequence prediction and unifies view feature extraction and fusion in a Transformer that models view and spatial relationships.

    Source: §I · pp. 1–2

  • Results

    With 30% of the parameters of other recent CNN-based methods, VolT achieves state-of-the-art multi-view reconstruction accuracy on ShapeNet.

    Source: §I · p. 2

  • Takeaways & Limitations

    The Transformer-based methods show better scaling ability as the number of input views increases, with EVolT's mean IoU rising by 0.04 from 8 to 24 views.

    Source: 1) Quantitative results · p. 6

Abstract

from arXiv · show

Deep CNN-based methods have so far achieved the state of the art results in multi-view 3D object reconstruction. Despite the considerable progress, the two core modules of these methods - multi-view feature extraction and fusion, are usually investigated separately, and the object relations in different views are rarely explored. In this paper, inspired by the recent great success in self-attention-based Transformer models, we reformulate the multi-view 3D reconstruction as a sequence-to-sequence prediction problem and propose a new framework named 3D Volume Transformer (VolT) for such a task. Unlike previous CNN-based methods using a separate design, we unify the feature extraction and view fusion in a single Transformer network. A natural advantage of our design lies in the exploration of view-to-view relationships using self-attention among multiple unordered inputs. On ShapeNet - a large-scale 3D reconstruction benchmark dataset, our method achieves a new state-of-the-art accuracy in multi-view reconstruction with fewer parameters ($70\%$ less) than other CNN-based methods. Experimental results also suggest the strong scaling capability of our method. Our code will be made publicly available.

I. INTRODUCTION

CNN-based multi-view reconstruction separates feature extraction from fusion and rarely explores relationships among views, while performance can saturate as input views increase. VolT unifies these stages in a Transformer and achieves state-of-the-art ShapeNet accuracy with fewer parameters than recent CNN methods.

I. INTRODUCTION

  • CNN approaches typically design feature extraction and view fusion separately, rarely explore cross-view object relations, and may saturate when input views exceed about four.RNN-based view modeling can be computationally inefficient and permutation-sensitive for unordered inputs.Source: §I · p. 1
  • VolT reformulates reconstruction as sequence-to-sequence prediction and unifies feature extraction and view fusion in a Transformer that jointly models view-view, view-volume, and volume-volume relationships.The encoder relates 2D views; the decoder relates input views to spatial queries and models correlations among 3D locations.Source: §I · pp. 1–2
  • Vanilla Transformer depth can degrade reconstruction accuracy as view embeddings converge, motivating a divergence-enhanced operation that preserves differences among views in deeper layers.The introduction describes this as addressing divergence decay and attention uniformity in multi-view reconstruction.Source: §I · pp. 1–2
  • 30% of the parameters of other recent CNN-based methods gives VolT a new state-of-the-art for multi-view reconstruction on ShapeNet.The method also shows better scaling capability as the number of input views increases.Source: §I · p. 2

A. Multi-view 3D Reconstruction

Prior multi-view reconstruction methods use distinct fusion strategies, with trade-offs in view association learning and permutation invariance. The paper presents a Transformer framework that encodes views, decodes spatial queries into a 3D volume, and has three implementations.

A. Multi-view 3D Reconstruction

  • CNN reconstruction systems commonly separate 2D-view encoding, 3D-volume decoding, and fusion; pooling misses view associations, while RNN fusion is computationally expensive and permutation-variant.Learnable weighted-sum fusion was introduced to address limitations of pooling-based methods.Source: A. Multi-view 3D Reconstruction · p. 2

B. Transformer

  • Transformer self-attention aggregates features across token pairs, offering a way to model deep relationships between views while retaining permutation-invariant processing of unordered inputs.The section contrasts this potential with the difficulty of designing CNN-based fusion that explores deep view relationships while remaining permutation-invariant.Source: B. Transformer · p. 3

A. Framework

  • The framework uses view-attention layers in a 2D-view encoder and volume-attention layers in a 3D-volume decoder, whose spatial queries produce volume predictions that are stitched into the reconstruction.The decoder also learns global correlations among spatial locations.Source: A. Framework · p. 3
  • VolT uses standard VGG16 view features, VolT+ uses embeddings from an advanced pretrained CNN, and EVolT adds view-divergence enhancement with a CNN shared across views.The authors prefer EVolT's smaller parameter size over an EVolT+ variant using advanced CNN features.Source: A. Framework · p. 3

B. Divergence-enhanced 2D-view Encoder

The divergence-enhanced 2D-view encoder refines view-shared CNN embeddings with self-attention while preserving the unordered nature of the input views. Its attention layers combine learned view-to-view relationships with a divergence-enhancing feature path.

B. Divergence-enhanced 2D-view Encoder

  • The encoder refines view-shared CNN embeddings across multiple views, omitting positional encodings to keep the view sequence permutation invariant.It stacks six blocks, each containing a multi-head divergence-enhanced view-attention layer and a position-wise feed-forward network.Source: B. Divergence-enhanced 2D-view Encoder · p. 3
  • Scaled dot-product attention learns view-to-view relationships, while DiView concatenates internal view features with input embeddings through skip connections.The final-layer embeddings serve as the 2D-view encoder output.Source: B. Divergence-enhanced 2D-view Encoder · p. 3
  • Each attention head projects the view embeddings into queries, keys, and values; the attention output is a score-weighted sum of values.The heads are concatenated and projected through Wview, and the block also applies a position-wise feed-forward network.Source: B. Divergence-enhanced 2D-view Encoder · pp. 3–4

C. 3D-volume Decoder

The 3D-volume decoder uses learnable volume queries to model relationships among spatial locations and between view and spatial domains. It processes these representations with attention and feed-forward layers, then projects and groups the volume outputs into a reconstruction.

C. 3D-volume Decoder

  • The decoder learns global correlations among spatial locations and relationships between view and spatial domains using learnable 3D-volume queries with positional encodings.Each decoder block contains volume attention, view-volume attention, and a feed-forward network.Source: C. 3D-volume Decoder · p. 4
  • Volume attention learns global dependencies among 3D volumes, while view-volume attention integrates information across view and spatial domains.The decoder uses its final-layer embeddings as output.Source: C. 3D-volume Decoder · p. 4
  • A linear function projects each output embedding into 3D space, after which the predicted volumes are reshaped and grouped into the final reconstruction.The decoder’s volume queries receive positional encodings to retain spatial position information.Source: C. 3D-volume Decoder · p. 4

A. Dataset

The evaluation uses a 13-category, 43,783-object ShapeNet subset and reports voxel-overlap and surface-distance metrics for reconstruction quality.

A. Dataset

  • ShapeNet evaluation uses 13 categories and 43,783 objects, with 24 rendered views per object.The dataset follows the same setting as 3D-R2N2.Source: A. Dataset · p. 4

1) IoU:

  • IoU measures agreement between predicted and ground-truth voxel grids, with higher values indicating better reconstruction.Predicted occupancy probabilities are binarized using an optimal fixed threshold for the compared methods.Source: 1) IoU · p. 4

2) F-Score:

  • F-Score, is the harmonic mean of precision and recall at distance threshold d; higher scores at stringent thresholds indicate better reconstruction.Precision counts reconstructed points within d = 1% of ground truth, while recall measures ground-truth points within d of the reconstruction.Source: 2) F-Score · p. 4

3) Divergence measurement for multi-view representations:

The section measures how multi-view attention and representations converge across layers, then reports implementation settings and model-size comparisons.

3) Divergence measurement for multi-view representations:

  • The divergence measure D uses Euclidean distances between view-to-view attention vectors; smaller D indicates greater similarity and convergence among view representations.Each row of the attention-score matrix represents one view's weights to the other views.Source: 3) Divergence measurement for… · p. 5

C. Implementation Details

  • Training uses batch size 64, 224 × 224 view images, and 32 × 32 × 32 voxel outputs, with AdamW for VolT and its variants.The optimizer uses β1 = 0.9 and β2 = 0.999.Source: C. Implementation Details · p. 5
  • EVolT uses around 30% of the parameters of Pix2Vox-A and Pix2Vox++/A, which use an additional 3D-CNN refiner for their reported best results.The paper's end-to-end methods do not require that additional refiner; VolT+ instead uses an advanced CNN feature extractor.Source: C. Implementation Details · p. 5

D. Multi-view 3D Object Reconstruction

On ShapeNet, EVolT leads the compared methods on 24-view IoU and F-Score, and the paper reports stronger scaling as view count increases. Qualitative examples also show more complete reconstructions.

D. Multi-view 3D Object Reconstruction

  • EVolT achieves the highest IoU and F-Score among compared methods in 24-view ShapeNet reconstruction.VolT performs between Pix2Vox-A and Pix2Vox++/A, while VolT+ improves on VolT but remains behind EVolT.Source: 1) Quantitative results · pp. 5–6
  • Figure 3 evaluates the view-divergence enhancement's effect on reconstruction results.The supplied passages identify the figure's purpose but do not report its comparison outcome.Source: Figure 3 · p. 6
  • EVolT's mean IoU increases by 0.04 from 8 to 24 views, eight times Pix2Vox++/A's improvement; the paper reports stronger scaling for its Transformer methods.The proposed methods achieve the best F-Score above 6 views and best IoU at higher view counts, though the supplied passage truncates the IoU threshold.Source: 1) Quantitative results · p. 6
  • Across 12-, 18-, and 24-view examples, EVolT produces more accurate, complete shapes, including chair legs and monitor stands missing from other methods.Figure 2 compares reconstructions from different numbers of input views.Source: Figure 2 · p. 6, 1) Quantitative results · p. 7

E. Ablation Study

The ablation reports that EVolT outperforms VolT, which in turn outperforms Pix2Vox-A, and that divergence enhancement preserves more view diversity in deeper layers. The conclusion reiterates ShapeNet state-of-the-art accuracy with fewer parameters and identifies interpretability as future work.

1) Effect on 3D reconstruction accuracy:

2) Effect on the view divergence:

  • In deeper layers, EVolT retains more diverse view attention and higher D values than VolT, whose D declines with encoder depth.A small D indicates greater convergence among multi-view representations.Source: 2) Effect on the view divergence · pp. 7–8
  • The ablations indicate that view-divergence enhancement is essential to improving EVolT performance and relieving convergence among multi-view representations.Source: 2) Effect on the view divergence · p. 8

V. CONCLUSION AND FUTURE WORKS

  • The paper reports ShapeNet state-of-the-art accuracy with fewer parameters than CNN-based methods and plans future work on interpreting the framework and its view correspondences.The planned work includes visualizing and understanding correspondence between latent representations and input views.Source: §V · p. 8

I. COMPARISON OF STRUCTURES IN COMPETING METHODS

The section frames a comparison of Transformer-based and CNN-based reconstruction methods by their components and structure. It also examines view-representation convergence across attention layers using D-value distributions for VolT and EVolT.

I. COMPARISON OF STRUCTURES IN COMPETING METHODS

  • Table 6 compares the proposed Transformer-based reconstruction methods and existing CNN-based methods by components and structure.Source: §I · p. 9
  • Figure 6 presents a comparison of CNN-based methods and the proposed Transformer-based methods.Source: Figure 6 · p. 9

A. View Divergence

  • EVolT's D-value density has greater variance than VolT's and retains a wider range of D values in deeper attention layers, indicating slower convergence degradation.A small D denotes greater convergence among multi-view representations; Figure 7 plots KDE-estimated D-value densities across layers.Source: A. View Divergence · pp. 9–10, Figure 7 · p. 10

B. Qualitative Results

The qualitative comparison covers reconstructions of several object categories using different numbers of input views. It reports superiority for the proposed method in reconstruction topology and detail.

B. Qualitative Results

Related papers

Loading 2103.12957v1…