Source-linked AI summary
TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
Aljaž Božič, Pablo Palafox, Justus Thies, Angela Dai, Matthias Nießner
TL;DR
Monocular video reconstruction must recover complete 3D geometry despite challenging frame selection and the computational cost of processing long sequences. TransformerFusion learns attention-based, coarse-to-fine feature fusion for online reconstruction, and reports state-of-the-art performance across competing reconstruction approaches. Its main limitations are severe occlusion, partial observation, and transparent objects.
Problem
Dense, complete monocular-video reconstruction remains challenging for interactive settings, and processing all frames jointly becomes computationally expensive for long videos or large scenes.
Method
TransformerFusion incrementally unprojects encoded RGB-frame features into a 3D volume, uses transformer attention to fuse informative views, and decodes coarse-to-fine features into surface occupancy.
Results
TransformerFusion achieves state-of-the-art reconstruction performance, with coarse-to-fine filtering improving fusion speed from 2 FPS to 7 FPS.
Takeaways & Limitations
Learned view selection focuses reconstruction on informative image features, while coarse-to-fine fusion supports interactive online scanning.
Takeaways & Limitations
Severe occlusions, partial observations, and transparent objects can cause missing or inaccurate reconstructed geometry.
Abstract
from arXiv · showhide
We introduce TransformerFusion, a transformer-based 3D scene reconstruction approach. From an input monocular RGB video, the video frames are processed by a transformer network that fuses the observations into a volumetric feature grid representing the scene; this feature grid is then decoded into an implicit 3D scene representation. Key to our approach is the transformer architecture that enables the network to learn to attend to the most relevant image frames for each 3D location in the scene, supervised only by the scene reconstruction task. Features are fused in a coarse-to-fine fashion, storing fine-level features only where needed, requiring lower memory storage and enabling fusion at interactive rates. The feature grid is then decoded to a higher-resolution scene reconstruction, using an MLP-based surface occupancy prediction from interpolated coarse-to-fine 3D features. Our approach results in an accurate surface reconstruction, outperforming state-of-the-art multi-view stereo depth estimation methods, fully-convolutional 3D reconstruction approaches, and approaches using LSTM- or GRU-based recurrent networks for video sequence fusion.
1 Introduction
TransformerFusion addresses monocular 3D reconstruction from RGB video with learned transformer feature fusion, coarse-to-fine processing, and interactive online reconstruction. It learns to select informative views for each 3D location while improving reconstruction performance and runtime.
- Motivation: Monocular 3D reconstruction seeks complete, accurate object or environmental geometry from 2D RGB observations and supports navigation, interaction, editing, and scene understanding.Geometric reconstructions also support 3D object detection and semantic and instance segmentation.
- Motivation: Dense, complete reconstruction of large-scale environments from monocular video remains challenging for interactive systems, while equal-weight frame averaging can include less informative observations.The passage identifies motion blur, rolling-shutter artifacts, glancing views, and partial views as sources of reduced information.
- Outcome: The method is presented as achieving state-of-the-art reconstruction results through transformer-based attention over image views.Figure 1 visualizes view attention for the most relevant frame at each 3D location.
- Approach: TransformerFusion fuses RGB video frames into a 3D scene representation and learns to attend to the most informative image features for each 3D location.The transformer incrementally fuses features extracted from observed images and identifies relevant views for local reconstruction.
- Approach: A coarse-to-fine hierarchy enables online reconstruction at interactive frame-rates while retaining fine-level processing where needed.The paper presents coarse-to-fine fusion as contributing to both reconstruction performance and runtime.
2 Related Work
Prior work estimates depth from multiple views, fuses volumetric information, predicts surface occupancy directly, and applies transformers across vision tasks. TransformerFusion extends transformer-based processing to feature fusion for monocular-video 3D scene reconstruction.
- Multi-view depth estimation: COLMAP established patch matching for multiview stereo, but dense reconstruction is difficult in regions lacking distinctive color features.Floor and walls are cited as examples of such regions.
- Monocular RGB reconstruction: Monocular reconstruction systems combine multi-view depth estimation with volumetric fusion, but noisy depth estimates can introduce artifacts.This motivated approaches that directly predict 3D surface reconstruction instead of per-frame depth estimates.
- Monocular RGB reconstruction: SurfaceNet predicts 3D surface occupancy from two RGB images by converting volumetrically averaged colors with a 3D convolutional network.This represents a direct surface-prediction alternative to fusing noisy per-frame depths.
- Transformers in computer vision: Transformers have been applied to tasks including object detection, video classification, image classification, image generation, and human reconstruction.TransformerFusion applies the architecture to feature fusion for 3D scene reconstruction from monocular video.
3 End-to-end 3D Reconstruction using Transformers
TransformerFusion reconstructs scene geometry by fusing coarse and fine image features into spatial 3D grids, refining them, and decoding interpolated features into an occupancy field. Coarse-to-fine filtering focuses computation near surfaces, reducing processing requirements while supporting interactive reconstruction.
- The method encodes each RGB image into coarse and fine features, then uses separate transformers to fuse them at sampled 3D locations.Coarse and fine samples are taken at 30 cm and 10 cm resolutions, respectively.
- 3D convolutional refinement networks improve the spatially fused coarse and fine feature grids before surface prediction.Separate refinement networks operate at the coarse and fine spatial resolutions.
- A lightweight MLP decodes trilinearly interpolated coarse and fine features into occupancy values for the final surface reconstruction.Occupancy values o ≥0.5 represent occupied points, while lower values represent free space; the model is trained end-to-end with BCE losses.
- Coarse-to-fine filtering improves reconstruction performance and enables interactive runtime by avoiding unnecessary high-resolution surface extraction.The paper reports that the filtering mechanism produces a runtime speed-up and focuses model capacity on surface geometry.
- The transformers produce fused features and attention weights, allowing the system to retain informative views for each 3D location during incremental reconstruction.The model processes temporal feature inputs and returns attention weights used for view selection.
- Near-surface masks filter free-space regions so expensive fine-scale processing is performed only where coarse and fine masks exceed 0.5.This sparsifies the volume and focuses the surface prediction network on regions close to the surface.
4 Experiments
Experiments evaluate reconstruction metrics, state-of-the-art comparisons, design ablations, view-count effects, attention-based selection, and failure cases. Transformer fusion, spatial refinement, and coarse-to-fine filtering improve reconstruction quality, while severe occlusions and partial observations remain challenging.
- Evaluation setup: The evaluation measures geometric accuracy, completion, Chamfer distance, precision, and recall using sampled mesh points and an occlusion mask.Accuracy measures predicted-to-ground-truth error, completion measures the reverse direction, and Chamfer averages the two.
- Comparison with state of the art: TransformerFusion outperforms existing methods, including multi-view depth estimation, recurrent fusion, and direct 3D surface reconstruction approaches.DeepVideoMVS performs best among the compared multi-view depth methods, while NeuralRecon and Atlas improve further through direct surface prediction, but TransformerFusion is reported to outperform all existing methods.
- Ablations: Learned transformer fusion significantly improves performance over independently processed views combined by averaging or MLP-predicted weights.The transformer learns to attend to informative views for each spatial location rather than relying on averaging-based alternatives.
- Ablations: Spatial refinement produces more spatially consistent and complete reconstructions, while removing it considerably worsens geometry completion and recall.The refinement aggregates feature information in the spatial domain.
- Ablations: Coarse-to-fine filtering improves performance and increases fusion speed from 2 FPS to 7 FPS, approximately a 3.5× speed-up.Predicting coarse and fine near-surface masks focuses computation on surface geometry.
- Ablations: Using K = 16 views performs better than sparser K = 8 or K = 4 settings, while limiting views reduces execution time and memory consumption.The experiments find that all 16 views contribute to the feature for each 3D grid location.
- Ablations: Attention-based frame selection outperforms random selection, with the performance gap increasing when fewer views are fused.Random selection causes noticeable drops in Chamfer and F-score and is more likely to miss important views at K = 8 or K = 4.
- Limitations: Severe occlusions and partial observations can prevent accurate reconstruction of details such as chair legs, monitor stands, and bookshelves.Transparent objects such as frameless glass windows are often reconstructed as empty space.
5 Conclusion
TransformerFusion introduces online transformer-based feature fusion for monocular 3D scene reconstruction from RGB views. Its coarse-to-fine fusion improves reconstruction performance and runtime, while learned view selection supports state-of-the-art reconstruction performance.
- Conclusion: TransformerFusion performs online feature fusion from RGB input views using a transformer-based approach for monocular 3D scene reconstruction.The method learns to exploit informative image-view features for geometric reconstruction.
- Conclusion: Coarse-to-fine transformer feature fusion improves effective reconstruction performance and runtime.
- Conclusion: The learned fusion exploits the most informative image-view features and achieves state-of-the-art reconstruction performance.
A Additional Results
Additional results visualize transformer-based spatial and temporal ablations, compare transformer-input choices, and examine attention-based frame selection. They show that projected depth and view rays help feature fusion, while random selection becomes relatively worse when fewer views are available.
- View selection: Figure 4 visualizes selected camera views and their attention weights for specific 3D locations in a ScanNet reconstruction.Color temperature encodes attention weight for the rendered selected views.
- Additional ablations: Qualitative ablations on ScanNet visualize the effects of spatial refinement and temporal transformer feature fusion using RGB input.The ground truth is reconstructed using the input depth, while each method receives only RGB input.
- Transformer inputs: Projected depth and view-ray inputs help the transformer fuse features more effectively for 3D reconstruction.
- View selection: The reconstruction-quality gap between attention-based and random view selection increases as fewer views can be selected.Random selection is more likely to miss important views when the available selection is sparse.
B Reproducibility
The reproducibility materials document the network variants, repeated experiments, and runtime behavior of the coarse-to-fine reconstruction pipeline.
- Runtime: Runtime analysis reports separate per-frame feature-fusion and per-chunk mesh-extraction measurements.
- Runtime: Surface reconstruction can run after feature fusion or incrementally every few frames for interactive feedback.
- Runtime: Feature fusion reaches 7.66 FPS with 16 views, 10.17 FPS with 8 views, and 12.28 FPS with 4 views.
- Network architecture: Independent coarse and fine feature-fusion and refinement networks implement the two hierarchy levels.
- Reproducibility: The approach and ablations were run three times, with F-score means and standard deviations reported using error bars.
C Limitations
The approach is limited when observations are incomplete or surfaces are transparent. Occluded regions may lack detail, while transparent surfaces can be reconstructed as free space.
- Partially observed objects with large occluded regions can have missing occluded parts or inaccurate small details.
- Transparent objects such as glass windows are often labeled as free space instead of reconstructed as surfaces.
D Data
The method is trained and evaluated on ScanNet, which contains anonymized static indoor environments. Qualitative test-set results are shown for representative ScanNet scenes.
- ScanNet provides static indoor environments, with anonymized scenes and locations collected under a noncommercial academic license.
- Representative scenes from the ScanNet test set are used for qualitative results.