Revealing Scenes by Inverting Structure from Motion Reconstructions

1 University of Florida 2 Microsoft Research

• Figure 1 shows a detailed image reconstructed from an SfM point cloud that closely resembles its source image.The figure presents a top-view reconstruction, points projected into the source-image viewpoint, the reconstructed image, and the source image.Source: Figure 1 · p. 1

• Always-on devices enabled by ARCore, ARKit, 3D mapping APIs, and HoloLens use image-based localization, which requires persistent storage of SfM point-cloud models [38].Source: §1 · p. 1
• SfM point clouds and attributes such as color and SIFT descriptors can reveal detailed, comprehensible scene images after source images are discarded, creating an overlooked privacy risk.Source: §1 · p. 1
• The attack targets sparse SfM models without source images and must handle missing keypoint orientation and scale, descriptor image sources, and 3D point visibility.Source: §1 · pp. 1–2
• A three-network cascade estimates visibility, reconstructs the image, and uses an adversarial framework to refine image quality.Source: §1 · p. 2
• The authors analyze attacks using descriptors, color, camera poses, and visibility, report vulnerability even for descriptors alone, and show that arbitrary novel views can enable virtual tours.Source: §1 · p. 2
• The cascade is trained on 700+ indoor and outdoor SfM reconstructions from 500k+ NYU2 and MegaDepth [24] images, with data and visibility labels generated using COLMAP.Source: §1 · p. 2

Inverting features

• Unlike feature-inversion work [20] [29] [9] [8], this work targets SfM-stored SIFT despite sparse irregular points, missing keypoint scale and orientation, unknown descriptor sources and perspective distortions, and unknown visibility.Each 3D point typically has one descriptor sampled from an arbitrary image, which can introduce photometric inconsistencies.Source: §2 · p. 2

Image-to-Image Translation

• Unlike category-specific image-translation methods such as Pix2Pix [19], CycleGan, and Contour2Im [5], these networks generalize to arbitrary images using large-scale indoor and outdoor SfM datasets.The approach shares encoder-decoder, U-Net, adversarial-loss, and perceptual-loss building blocks with prior translation methods.Source: §2 · p. 2

Upsampling

• Unlike prior upsampling methods focused on uniform sparsity, this approach addresses nonuniformly sampled, high-dimensional, noisy inputs whose descriptors come from different images and viewpoints.Source: §2 · p. 2

Novel view synthesis and image-based rendering

• The approach claims photorealistic visual quality from sparse SfM reconstructions, presenting a privacy risk while also offering potential for image generation from 3D reconstructions.Prior deep-network work improved free-viewpoint image-based rendering [12] and explored monocular depth estimation and novel-view synthesis with U-Nets [11] [24] [31].Source: §2 · p. 2

CNN-based privacy attacks and defense techniques

• After CNN attacks on image obfuscation [30] [44] and learned defenses [33] [10] [13], concurrent work proposed obfuscated-map camera localization [41] to defend against this inversion attack.Source: §2 · pp. 2–3

3. Method

• The pipeline projects viewpoint-conditioned SfM points and attributes into a 2D feature map, then cascades visibility estimation, coarse RGB reconstruction, and refinement into the final image.When multiple points project to one pixel, it retains the closest point’s attributes and depth.Source: §3 · p. 3

3.1. Visibility Estimation

• COARSENET’s implicit visibility reasoning was inaccurate in regions with few visible relative to occluded points; VISIBNET predicts visibility and culls occluded points, significantly improving reconstructed-image quality, especially there.VISIBSPARSE uses a 3×3 min filter and retains entries whose depth is no greater than 5% of the filtered depth.Source: §3.1 · p. 3

3.2. Architecture

• The three subnetworks use U-Nets with skip connections; extra decoder convolutions spread low-level SIFT information across output pixels and reduce artifacts from sparse, irregular inputs.The input feature map can combine depth, color, and SIFT features.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3. Optimization

• Training proceeds separately: VISIBNET uses automatic VISIBDENSE masks, COARSENET combines L1 pixel and VGG16 perceptual losses, and REFINENET adds adversarial loss while earlier networks stay fixed.REFINENET uses a conditional discriminator to distinguish synthesized images from source images.Source: §3.3 · pp. 4–5

4. Experimental Results

• Qualitative results show reconstructions across varied MegaDepth and NYU scenes, handling sparse, irregular point distributions and missing information, including household interiors and an aerial scene.NYU scenes generally produce fewer SfM points than MegaDepth scenes; moving objects are not recovered because the method relies on SfM.Source: §4 · p. 5, Figure 3 · p. 6

4.1. Visibility Estimation

• The depth-only VISIBNET reaches 94.8% mean classification accuracy on MD and 89.2% on NYU, even with only 20% of input samples.Removing points VISIBNET predicts are occluded consistently improves COARSENET over implicit visibility handling, with visual examples showing artifacts removed.Source: Table 2 · p. 5, §4.1 · p. 6, Figure 5 · p. 7

4. Experimental Results

• In single-image SIFT inversion, COARSENET remains robust to unknown keypoint scale and orientation, while multiple image sources make inversion harder but still yield recognizable results.The study tested four variants at three sparsity levels; results remained reasonably robust across the tested sparsities.Source: §4 · p. 5, Figure 4 · p. 7

4.2. Relative Significance of Point Attributes

• COARSENET captures fine details from depth-only inputs across sparsity levels, while adding color and SIFT descriptors significantly improves visual quality.The point-attribute comparison is intended to assess which optional SfM attributes might be removed when storing a model for privacy.Source: §4.2 · p. 7, Table 4 · p. 8, Figure 6 · p. 8

4.3. Significance of RefineNet

• When color is omitted, REFINENET can recover plausible colors from depth and descriptors, and it also improves COARSENET results when all attributes are available.The reported examples indicate that removing color alone may not preserve privacy because REFINENET can still produce plausible color reconstructions.Source: §4.3 · p. 7, Figure 7 · p. 8

4.4. Novel View Synthesis

• The method synthesizes realistic novel views that can form virtual tours and help attackers interpret cluttered scenes, even when synthesized images contain artifacts.The paper presents qualitative novel-view examples for two NYU scenes.Source: §4.4 · p. 8, Figure 8 · p. 8

5. Conclusion

• The authors introduce sparse SfM point-cloud inversion and demonstrate that limited information stored with 3D models can produce surprisingly high-quality scene images.They identify the vulnerability as a privacy and security risk of storing point clouds whose contents may reveal the underlying scene’s appearance.Source: §5 · p. 8
• Concurrent work on privacy-preserving camera pose estimation [41] proposes a defense against the attacks investigated in this paper.The authors call for privacy-preserving point-cloud representations and camera-localization techniques that cannot easily be inverted to reveal scene appearance.Source: §5 · p. 8
• The authors identify privacy-preserving features for recovering image-to-3D-model correspondences as a future research direction.Source: §5 · p. 8

A. Implementation Details

• The supplementary material describes the network architecture and training procedure in greater detail.Source: A. Implementation Details · p. 11

A.1. Architecture

• VISIBNET, COARSENET, and REFINENET each use an encoder-decoder U-Net with skip connections, and pass the decoder output through three convolutional layers to produce the final output.The input feature map concatenates selected depth, color, and SIFT features at valid 2D sample locations.Source: §A.1 · p. 11
• The encoder comprises six stride-2 convolutional layers with 256, 256, 256, 512, 512, and 512 kernels, respectively.Each layer uses 4×4 kernels, bias, batch normalization, and ReLU.Source: §A.1 · p. 11
• The decoder uses six upsampling convolutional layers followed by 128-, 64-, 32-, and 3-kernel layers; REFINENET uses leaky ReLU, while VISIBNET outputs one channel.The final decoder layer uses tanh instead of ReLU; REFINENET replaces decoder ReLUs with leaky ReLUs.Source: §A.1 · p. 11
• The REFINENET adversarial discriminator combines six convolutional layers with 256, 256, 256, 512, 512, and 512 kernels, followed by fully connected layers of 1024, 1024, 1024, and 2 nodes.Convolutional layers use 3×3 kernels, max pooling, batch normalization, and leaky ReLU; the final fully connected layer uses softmax.Source: §A.1 · p. 11

A.2. Optimization

• All networks were trained with Adam (β1 = 0.9, β2 = 0.999, ϵ = 1e−8; learning rate 0.0001) on 256 × 256 inputs with Xavier initialization.For training, images were resized so the smaller dimension was randomly set to 296, 394, or 512 pixels, then randomly cropped to 256 × 256. The trained network processed images at 512 × 512 resolution.Source: §A.2 · p. 11

B. Additional Results

• For two scenes with available SfM camera poses, the method reconstructs source video frame by frame and synthesizes novel-view images for virtual tours.The supplementary video presents these results and shows that novel viewpoints can reveal scene appearance, layout, and geometry.Source: B. Additional Results · p. 11
• With only 20% of the projected 3D points, synthesized images remain largely recognizable; results are also shown at 60% and 100%.The examples use the complete pipeline, and the text notes that outputs remain interpretable despite high simulated 2D sparsity.Source: B. Additional Results · p. 11, Figure A1 · p. 12
• Failure cases include visibility errors, artifacts around lines and occlusions, blurry edges from low sample density, and hallucinated structures from spurious points.Dense background points can cause a foreground statue base to disappear; parallel lines may be poorly handled, and straight lines may appear curved or bent.Source: Figure A2 · p. 12