PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction

Bo Yang, Sen Wang, Andrew Markham, Niki Trigoni

arXiv:1808.00758v2cs.CVcs.AIcs.LGcs.RO

TL;DR

Multi-view 3D reconstruction needs aggregation methods that handle variable image sets without permutation sensitivity or losing useful feature information. The paper introduces AttSets and FASet for attentive, permutation-invariant feature aggregation, and reports that they significantly outperform existing approaches. Its experiments support robustness and efficiency for multi-view 3D reconstruction, but generality beyond this task remains untested.

  • Problem

    Existing recurrent and pooling-based aggregation methods struggle with permutation consistency, long-term refinement, efficient processing, and dynamic numbers of input images.

  • Method

    AttSets attentively aggregates arbitrary-sized deep feature sets, while FASet decouples feature learning from attention learning to support variable-sized inputs.

  • Results

    AttSets with FASet significantly outperforms existing aggregation approaches across quantitative and qualitative multi-view 3D reconstruction experiments.

  • Takeaways & Limitations

    AttSets and FASet provide permutation-invariant, computationally efficient, and robust aggregation for multi-view 3D reconstruction with variable input-set sizes.

  • Takeaways & Limitations

    The experiments focus exclusively on multi-view 3D reconstruction, leaving generality to other set-based tasks untested.

Abstract

from arXiv · show

We study the problem of recovering an underlying 3D shape from a set of images. Existing learning based approaches usually resort to recurrent neural nets, e.g., GRU, or intuitive pooling operations, e.g., max/mean poolings, to fuse multiple deep features encoded from input images. However, GRU based approaches are unable to consistently estimate 3D shapes given different permutations of the same set of input images as the recurrent unit is permutation variant. It is also unlikely to refine the 3D shape given more images due to the long-term memory loss of GRU. Commonly used pooling approaches are limited to capturing partial information, e.g., max/mean values, ignoring other valuable features. In this paper, we present a new feed-forward neural module, named AttSets, together with a dedicated training algorithm, named FASet, to attentively aggregate an arbitrarily sized deep feature set for multi-view 3D reconstruction. The AttSets module is permutation invariant, computationally efficient and flexible to implement, while the FASet algorithm enables the AttSets based network to be remarkably robust and generalize to an arbitrary number of input images. We thoroughly evaluate FASet and the properties of AttSets on multiple large public datasets. Extensive experiments show that AttSets together with FASet algorithm significantly outperforms existing aggregation approaches.

1 Introduction

The paper addresses limitations of recurrent and pooling-based aggregation for multi-view 3D reconstruction by introducing AttSets, a permutation-invariant feed-forward attention module, and FASet, a training algorithm for dynamic-sized feature sets. Experiments on multiple public datasets show consistent improvements over existing aggregation approaches.

  • Motivation: Deep learning methods estimate dense 3D shapes from multiple images, but GRU-based fusion is permutation variant, struggles with long-term dependencies, and requires sequential processing.These limitations make predictions sensitive to image order, hinder refinement with additional images, and reduce computational efficiency.
  • Motivation: Pooling methods avoid recurrent limitations but capture only max, mean, or summed values and are not robust to dynamically sized input sets.The paper characterizes these operations as hard attentive because they do not learn to preserve useful information selectively.
  • AttSets: AttSets is a feed-forward module that learns attention scores for feature-set elements, weights their features, and sums them into fixed-size representations for 3D reconstruction.It can aggregate an arbitrary number of deep feature elements and be included in existing reconstruction networks.
  • FASet: FASet decouples encoder-decoder learning from attention-score learning, making AttSets-based networks robust and general to arbitrary numbers of input images.The method uses a two-stage training design and targets dynamically sized feature sets, unlike existing attention works limited to fixed-sized inputs.
  • Contributions and evaluation: AttSets is presented as permutation invariant and computationally efficient, while experiments demonstrate consistent improvement over existing aggregation approaches on multiple public datasets.The reported evaluation covers 3D object reconstruction from either single or multiple views.

2 Related Work

Prior multi-view reconstruction methods use recurrent or pooling-based feature fusion, while set-learning and attention methods address permutation invariance with important limitations. The paper positions AttSets as a simpler feed-forward attention module, complemented by FASet for robustness and generalization to arbitrarily sized deep sets.

  • Multi-view 3D Reconstruction: GRU-based fusion in 3D-R2N2 and LSM is permutation variant and inefficient for aggregating long image sequences.These methods recurrently fuse deep features from multiple images.
  • Multi-view 3D Reconstruction: Max, average, and TSDF-averaging methods preserve limited statistics or lose information during multi-view and depth-image fusion.SilNet and DeepMVS use max pooling, RayNet uses average pooling, and TSDF-based methods average signed-distance values.
  • Deep Learning on Sets: Set-learning methods require permutation-invariant functions that process arbitrary set sizes, but first- and higher-order pooling ignore much set information.First-order poolings lack trainable parameters, whereas higher-order poolings have additional limitations described in the paper.
  • Attention Mechanism: Coupled attention mechanisms are permutation variant and computationally time-consuming, whereas decoupled attention modules remove recurrence but may still aggregate variable-sized sets imperfectly.The related work contrasts recurrent attention with Transformer-style and other decoupled attention approaches.
  • Attention Mechanism: AttSets is a simplified feed-forward attention module without RNN coupling or the additional gating mechanism required by a concurrent approach.The paper also introduces FASet, described as enabling robust generalization to arbitrarily sized deep sets and as the first algorithm to improve robustness of feed-forward attention mechanisms.

3 AttSets

AttSets aggregates an arbitrary number of deep feature vectors into a fixed-dimensional output by learning feature-wise attention and summing weighted features. Its shared-function design makes the output permutation invariant and supports fully connected, 2D convolutional, and 3D convolutional implementations.

  • AttSets formulation: AttSets maps an arbitrary-sized feature set A to a fixed-dimensional vector y for a subsequent decoder.The input elements have fixed feature dimension D, while their number N may vary.
  • Attention aggregation: The module learns attention scores for each latent feature, uses them as a mask, and sums weighted features across set elements.Scores are produced by a shared function, normalized with softmax, multiplied by the original features, and summed.
  • Permutation invariance: AttSets is permutation invariant because its output coordinates use sums of permutation-equivariant terms in both numerator and denominator.Thus, reordering the deep feature set does not change y.
  • Implementation variants: The attention function g can use fully connected, 2D convolutional, or 3D convolutional layers, with shared weights preserving permutation invariance.Convolutional variants can use filters of size 1, 3, or many; they generally have fewer learnable parameters than the fully connected variant.

4 FASet

FASet separates optimization of the base encoder-decoder and AttSets attention module to prevent input-set-size bias. Its two-stage training is crucial for preserving robustness to dynamic input sets.

  • Motivation: Joint training makes encoded features and attention scores dependent on input-set size, preventing generalization to arbitrary numbers of images.Training on five images can perform well at five views yet fail with one image.
  • Motivation: FASet avoids the limitations of enumerating input-set sizes, which can produce mean features, reduce optimality, and make training impractical.The naive alternative jointly trains over various N values but cannot efficiently cover every possible input-set size.
  • FASet algorithm: Stage 1 optimizes the base encoder-decoder using single images, while Stage 2 optimizes AttSets using multiple input images.This gives the base encoder consistent derivatives and trains attention from multiple shared visual features.
  • FASet algorithm: FASet decouples the base network and AttSets parameters, optimizing them separately in two stages.The base network is optimized first, followed by the attention module, using the loss defined by the base network’s supervision signal.
  • Effect of FASet: Without FASet, feed-forward attention is ineffective for dynamic input sets; with it, AttSets retains superior robustness.After the base network converges, it need not be retrained because its optimization is agnostic to the attention module.

5 Evaluation

The evaluation tests AttSets and FASet within established 3D-R2N2 and SilNet encoder-decoder networks against recurrent, first-order pooling, and higher-order pooling alternatives. It assesses explicit voxel reconstruction on three datasets and implicit silhouette-based shape learning on Blobby using mean IoU.

  • Competing Approaches: The comparison includes GRU-based RNN aggregation, max/mean/sum first-order poolings, and higher-order BP, MHBN, and SMSO poolings under matched network and training settings.All competing approaches connect at the same encoder-decoder location as the proposed module.
  • Metrics: Explicit 3D reconstruction is evaluated on ShapeNetr2n2, ShapeNetlsm, and ModelNet40 using mean voxel-grid Intersection-over-Union.ShapeNetlsm changes camera viewing angles and lighting relative to the related dataset, enabling robustness and generality evaluation.
  • Metrics: Implicit 3D shape learning is evaluated on Blobby using mean IoU between predicted 2D silhouettes and ground truth.Each Blobby object provides five RGB images paired with viewing angles and silhouettes under varied lighting sources and texture models.
  • Experimental Protocol: The ShapeNetr2n2 experiments train all networks with one image in Stage 1, then use 2, 8, or 16 images in Stage 2 for AttSets training and competing-method fine-tuning.Tables 1–3 report the corresponding mean IoU results for the 13-category testing split.

5.1 Evaluation on ShapeNetr2n2 Dataset

On ShapeNetr2n2, AttSets with FASet consistently outperforms GRU and pooling approaches for single- and multi-view reconstruction while remaining more robust to varying numbers of input images. The experiments show that fixed-view training limits generalization, whereas random-view training improves robustness but does not match AttSets overall performance.

  • Training robustness: Training with a uniformly random number of views from 1 to 24 improves pooling robustness compared with fixed-view training, but performance remains intermediate across view counts.Random-view training improves results at N = 1 relative to 24-view training and at N = 24 relative to 1-view training, with the opposite tradeoff at the other endpoint.
  • Comparison settings: The reported LSM IoUs are excluded because LSM uses viewing angles and different rendered-image styles and train/test splits, making its scores not directly comparable.Its GRU aggregation module is nevertheless evaluated throughout these experiments.
  • Analysis: GRU produces reasonable shapes across view counts but saturates quickly after roughly 8 views because recurrent processing struggles with longer image sequences.The saturation is illustrated in the Group 4 results.
  • Analysis: Pooling approaches perform best near their training view count but generalize poorly to arbitrary counts, including few-view testing after training on 24 images.Pooled features from fewer test images are less representative than those formed from the larger training view set.
  • Analysis: AttSets clearly outperforms GRU and pooling approaches in both single- and multi-view 3D reconstruction and is more robust to variable input-view counts.FASet decouples base-network feature learning from AttSets attention-score learning, separating single-view reconstruction from multi-view aggregation.

5.2 Evaluation on ShapeNetlsm Dataset

On the ShapeNetlsm dataset, AttSets generalizes robustly to a previously unseen rendered-image style and outperforms competing approaches with both few and multiple input images.

  • Evaluation on ShapeNetlsm Dataset: AttSets based approach outperforms all others given either few or multiple input images on ShapeNetlsm.The evaluation uses synthesized ShapeNetlsm images corresponding to ShapeNet testing objects, whose rendered-image style and 3D labels were unseen during training.
  • Evaluation on ShapeNetlsm Dataset: The Baser2n2-AttSets approach shows better generality and robustness to new styles of rendered color images than other approaches.This result indicates that the approach does not overfit the training data.

5.3 Evaluation on ModelNet40 Dataset

On ModelNet40, all approaches are trained with the FASet two-stage strategy and evaluated across 1–12 input views under matched image permutations. Baser2n2-AttSets surpasses competing approaches by a large margin for both single- and multiple-view 3D reconstruction, consistently with earlier datasets.

  • Training protocol: All pooling-, GRU-, and AttSets-based networks are trained from scratch on ModelNet40 using the FASet algorithm for fair comparison.Stage 1 uses N = 1 until convergence; Stage 2 optimizes multi-view reconstruction.
  • Training protocol: Stage 2 trains networks either with N = 12 throughout or with N uniformly and randomly sampled from [1, 12] for each object.For Baser2n2-AttSets, the Stage 1 encoder-decoder is frozen and only AttSets is trained; competing approaches are fine-tuned.
  • Testing and results: Testing evaluates separately trained models at N = [1, 2, 3, 4, 5, 8, 12] using identical input-image permutations across approaches.Tables 8 and 9 report mean IoU for the two training groups, while Figure 13 presents Group 2 qualitative results.
  • Testing and results: Baser2n2-AttSets surpasses all competing approaches by a large margin for both single- and multiple-view 3D reconstruction.The results are consistent with previous experimental results on ShapeNetr2n2 and ShapeNetlsm datasets.

5.4 Evaluation on Blobby Dataset

On the Blobby dataset, the study compares AttSets-based, pooling-based, and GRU-based networks under the two-stage FASet training procedure. Models are tested with 1–4 input images, and results report mean IoU for predicted 2D silhouettes rather than 3D voxels.

  • Training setup: All compared networks use the two-stage FASet training algorithm, with GRU capacity matched to AttSets using one fully connected layer of 160 hidden units.Stage 1 trains with N=1; Stage 2 further optimizes networks for multi-view shape learning.
  • Training setup: Stage 2 trains separate groups with N=2 and N=4 images, freezing the trained base encoder-decoder while optimizing only AttSets in Basesilnet-AttSets for the N=2 group.Competing approaches are fine-tuned with the corresponding number of views, while the N=4 group trains all networks with four images.
  • Testing protocol: All networks trained in the two Stage 2 groups are separately tested with N=[1,2,3,4] using identical input-image permutations across methods.This testing protocol supports fair comparison across different aggregation approaches.
  • Results: Results are reported as mean IoUs for predicted 2D silhouettes, so they are not numerically comparable with prior experiments reporting IoU on 3D voxels.Table 10 and Table 11 contain the quantitative results, while Figure 14 presents qualitative Group 2 silhouette predictions.

5.5 Qualitative Results on Real-world Images · 5.6 Computational Efficiency

On manually collected Amazon images, the method demonstrates qualitative generality and permutation invariance, unlike the inconsistent GRU-based reconstructions. Under matched hardware and software conditions, AttSets is reported to be as efficient as pooling methods, with runtime measured across varying image counts.

  • 5.5 Qualitative Results on Real-world Images: Real-world Amazon shop images qualitatively demonstrate the generality of networks trained on synthetic ShapeNet2n2 data.The paper notes that no public real-world multi-view 3D reconstruction dataset is available, motivating manual image collection.
  • 5.5 Qualitative Results on Real-world Images: Across six image permutations per object, the GRU-based approach produces inconsistent 3D shapes and can lose object parts between permutations.Examples include recovering a chair arm or table leg in one permutation but failing to recover it in another.
  • 5.5 Qualitative Results on Real-world Images: The real-world evaluation qualitatively compares permutation invariance across different approaches.The comparison uses the same six permutations per object to test sensitivity to input-image ordering.
  • 5.6 Computational Efficiency: Baser2n2-AttSets and competing approaches are evaluated under the same Baser2n2 network, Titan X hardware, and software environment.The implementation uses Python 2.7, TensorFlow 1.2, CUDA 9.0, and cuDNN 7.1.
  • 5.6 Computational Efficiency: AttSets is reported to be as computationally efficient as pooling methods for single-object reconstruction.Table 12 reports mean time consumption in milliseconds for estimating a single 323 voxel grid from different numbers of images.
  • 5.6 Computational Efficiency: The computational evaluation measures average reconstruction time for a single 3D object given different numbers of input images.The reported measurements are expressed in milliseconds and use a 323 voxel grid.
  • 5.6 Computational Efficiency: The paper visualizes learned attention scores for deep feature sets using convolution-based AttSets.These learned scores are presented in Figure 17.

5.7 Comparison between Variants of AttSets · 5.8 Feature-wise Attention vs. Element-wise Attention · 5.9 Significance of FASet Algorithm

Sections 5.7–5.9 compare AttSets variants, feature-wise versus element-wise attention, and FASet against joint training. The experiments examine aggregation placement, attention granularity, and training strategy on ShapeNetr2n2.

  • 5.7 Comparison between Variants of AttSets: The study compares fully connected, 2D convolutional, and 3D convolutional AttSets variants under matched remaining architectures.The convolutional variants aggregate tensors within the 2D encoder or 3D decoder, respectively.
  • 5.7 Comparison between Variants of AttSets: fc and conv3d AttSets achieve similar mean IoU scores for single- and multi-view reconstruction on the ShapeNetr2n2 testing split.The result is reported for all 13 categories in Table 13.
  • 5.8 Feature-wise Attention vs. Element-wise Attention: AttSets is designed to learn feature-wise attention scores across the entire input deep feature set.The paper motivates this design as improving aggregation over dynamic feature sets evaluated earlier.
  • 5.8 Feature-wise Attention vs. Element-wise Attention: Element-wise attention assigns one scalar attention score to each input feature-set element, weighting all features within that element identically.The scores are softmax-normalized before weighted-summation pooling.
  • 5.8 Feature-wise Attention vs. Element-wise Attention: Feature-wise attention clearly outperforms element-wise attention in mean IoU for ShapeNetr2n2 3D reconstruction.The comparison uses the same Group 4 training settings and is reported in Table 14, with learned scores and predicted shapes illustrated in Figure 18.
  • 5.9 Significance of FASet Algorithm: FASet is evaluated against JoinT, which jointly optimizes base and attention parameters using a single loss.The comparison uses the same Group 4 training settings on ShapeNetr2n2.
  • 5.9 Significance of FASet Algorithm: The JoinT training approach tends to optimize the whole network according to its IoU scores in Table 15.The supplied passage introduces this comparison but does not provide the numerical scores.

6 Conclusion

The paper presents AttSets and FASet for aggregating deep feature sets in multi-view 3D reconstruction, with permutation invariance, computational efficiency, robustness, and flexible implementation. Quantitative and qualitative results show that AttSets significantly outperforms widely used aggregation approaches.

  • 6 Conclusion: AttSets and FASet aggregate deep feature sets for multi-view 3D reconstruction with permutation invariance, computational efficiency, robustness, and flexible implementation.The paper supports these properties with theory and extensive experiments.
  • 6 Conclusion: Quantitative and qualitative results show that AttSets significantly outperforms other widely used aggregation approaches.
Loading 1808.00758v2…