PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency

Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, Jitendra Malik

arXiv:1704.06254v1cs.CV

TL;DR

Single-view 3D reconstruction lacks direct 3D supervision in many real datasets, while traditional consistency methods require many views or instance-specific optimization. The paper introduces differentiable ray consistency, using expected costs over probabilistic ray events to train CNNs from multi-view observations. It reports improved single-view reconstruction on PASCAL VOC and encouraging controlled results, including robustness to noisy supervision.

  • Problem

    Single-view 3D prediction needs training signals when ground-truth 3D is unavailable, but traditional multi-view reconstruction generally requires many views and instance-specific optimization.

  • Method

    Differentiable ray consistency represents view consistency as expected event costs over probabilistic ray terminations, producing gradients for CNN-predicted shapes from depth, masks, color, or semantics.

  • Results

    The approach improves single-view object reconstruction on PASCAL VOC and shows encouraging controlled results using sparse multi-view observations, including robustness to noisy training signals.

  • Takeaways & Limitations

    Multi-view observations can supervise single-view 3D prediction without requiring ground-truth 3D models for every training example.

  • Takeaways & Limitations

    The formulation assumes known camera transformations across views and is currently applicable to voxel-occupancy representations.

Abstract

from arXiv · show

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) term. We show that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations e.g. foreground masks, depth, color images, semantics etc. as supervision for learning single-view 3D prediction. We present empirical analysis of our technique in a controlled setting. We also show that this approach allows us to improve over existing techniques for single-view reconstruction of objects from the PASCAL VOC dataset.

1. Introduction

The paper reframes shape–image consistency as a differentiable problem: rather than removing volume through space carving, it computes incremental shape changes from observations at arbitrary views. This differentiability connects 3D geometry with learning-based single-view reconstruction.

  • Space carving makes a shape consistent by removing volume along rays known not to intersect the object.
  • The paper seeks gradients that incrementally change a predicted 3D shape toward consistency with depth images, foreground masks, or color images.
  • The differential ray-consistency formulation is mathematically interesting independently of its applications.
  • Differentiability allows multi-view evidence to train neural networks for predicting a novel object’s 3D shape from one image.

2. Related Work

Prior work reconstructs shapes using explicit 3D supervision, image annotations, or multi-view optimization. This paper instead uses continuous consistency costs to provide learning signals from sparse observations and extend reconstruction to real data without ground-truth 3D.

  • Prior object-reconstruction approaches use morphable models, silhouettes, keypoints, or sparse shape skeletons for image-based supervision.
  • CNN-based reconstruction methods commonly require full 3D supervision and therefore primarily train on synthetically rendered data.
  • The paper relaxes explicit 3D supervision, enabling reconstruction beyond synthetic settings on real datasets without 3D annotations.
  • Geometry-based multi-view methods reconstruct particular objects by optimizing discrete variables for consistency across many views.
  • The proposed continuous cost functions provide CNN gradients through one computation and can use sparse, potentially single-view observations per instance.
  • Earlier work also learns single-view depth from stereo or multi-view reprojection inconsistency, motivating related supervision without direct depth targets.

3. Formulation

The formulation converts image-wide consistency into differentiable per-ray expected costs over probabilistic ray-termination events. Event costs can represent depth, masks, color, and semantics, producing gradients that reduce probabilities of observation-inconsistent events.

  • 3.1. View Consistency as Ray Consistency: The framework defines a view-consistency loss over an observation-camera pair and reduces it to differentiable consistency terms for individual rays.
  • 3. Formulation: The shape is represented by per-voxel emptiness probabilities in a traceable discretized grid, which need not be uniform.
  • 3.5. Incorporating Additional Labels: Optional per-voxel predictions extend the loss to auxiliary observations such as semantics and color through generalized event-cost terms.
  • 3.1. View Consistency as Ray Consistency: Each image pixel determines a world-space ray from camera intrinsics and extrinsics, with observations such as depth, color, foreground, or semantic labels.
  • 3.2. Ray-tracing in a Probabilistic Occupancy Grid: A ray induces termination events at traversed voxels or an escape event, with probabilities determined by preceding empty voxels and the terminating voxel.
  • 3.3. Event Cost Functions: Event costs compare predictions induced by each termination event with the associated ray observation; depth uses distance discrepancy, while masks encode intersection versus escape.
  • 3.3. Event Cost Functions: Foreground-mask consistency can be related to geometrically derived reprojection losses while supporting more general observations.
  • 3.4. Ray-Consistency Loss: The per-ray loss is the expected event cost, and its derivatives adjust voxel probabilities so inconsistent events become less likely.

4. Learning Single-view Reconstruction

A parameterized CNN predicts voxel occupancies, optionally with per-voxel auxiliary outputs, from one image. During training, view-consistency losses compare that prediction with additional observations of the same instance, while testing uses only the single image.

  • The CNN learns single-image 3D prediction without requiring ground-truth voxel models for training.
  • Training images are paired with one or more additional observations from other views, which may take different forms supported by the framework.
  • At test time, the learned CNN predicts a 3D shape from only one 2D image.
  • The network output contains voxel occupancy probabilities and optional per-voxel predictions for exploiting color or semantic observations.
  • Training minimizes inconsistency between each prediction and its corresponding observation-camera pairs using the differentiable loss.
  • About 1000 randomly sampled rays per view are used in each SGD iteration instead of all rays for faster training.

5. Experiments

Experiments evaluate DRC for object and scene reconstruction under varied supervision, view counts, and noise. The method performs competitively with baselines and extends single-view reconstruction to real images and ego-motion sequences.

  • 5.1. Empirical Analysis on ShapeNet: On ShapeNet, experiments use rendered single-view inputs and multi-view supervision across airplanes, cars, and chairs, with voxel mean IoU as the metric.Control settings include ground-truth 3D supervision, DRC with masks or depth, and depth fusion.
  • 5.1. Empirical Analysis on ShapeNet: DRC with a small set of multi-view observations approaches the upper bound from ground-truth 3D supervision.With perfect depth, DRC and depth fusion are comparable; DRC is more robust when the depth training signal is noisy.
  • 5.1. Empirical Analysis on ShapeNet: Performance initially improves as training views increase, then saturates after a few views; depth is more informative than masks with very few views.Mask supervision is comparable to depth overall but struggles with chair-seat concavities and limited-view settings.
  • 5.2. Object Reconstruction on PASCAL VOC: On PASCAL VOC, DRC trained only with PASCAL3D data performs comparably to a baseline using similar training data, while joint ShapeNet and PASCAL training improves across categories.The PASCAL learning signals use approximate pose and segmentation annotations, and improvements persist despite their imperfections.
  • 5.3. 3D Scene Reconstruction from Ego-motion: In Cityscapes, DRC produces coarse 3D occupancy and semantic predictions that render disparity and semantics under simulated forward motion.The representation captures structure hidden in the input image, such as cars occluding other cars, but moving objects and limited scene coverage remain challenges.
  • 5.4. Object Reconstruction from RGB Supervision: RGB-supervised ShapeNet results show that the learned model can infer shape and color, including chair concavities and shading for hidden parts.The paper notes artifacts below cars as an example of an error mode.

6. Discussion

The paper presents differentiable 3D–2D consistency for single-view reconstruction while identifying representation, camera-pose, and evaluation challenges that remain.

  • The formulation supports learning single-view reconstruction across multiple scenarios.
  • The current formulation applies to voxel-occupancy representations, motivating extensions to finer-grained alternatives.
  • The method assumes known camera transformations between views.
  • Ground-truth 3D can be bypassed during training, but benchmark data remains necessary for evaluation.

Appendix : Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency

The appendix identifies the paper’s authors, their Berkeley affiliation, and their contact addresses.

  • The paper is authored by Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik.
  • The authors are affiliated with the University of California, Berkeley.
  • The appendix lists author email addresses using the eecs.berkeley.edu domain.

A1. Gradient Derivations

The appendix restates the event-probability and ray-consistency equations, expands and simplifies them, then derives gradients with respect to predictions x.

  • The appendix re-iterates the equations for event probabilities and ray consistency loss.
  • Equation 12 is expanded using Equation 11.
  • The expanded expression is simplified to a final form.
  • The appendix computes derivatives of the ray consistency loss with respect to predictions x.

A2.1. Formulation

The appendix explains how ray-consistency costs are instantiated for masks, depth, RGB, and semantics, including treatment of rays that escape the modeled volume. It connects mask supervision to geometrically derived reprojection and identifies differentiability as the key requirement for usable observations.

  • Foreground-mask supervision minimizes discrepancy between the observed mask and reprojected occupancies.
  • The mask formulation is similar to prior learned or fixed reprojection functions for object reconstruction.
  • Depth supervision assigns escaping rays a fixed large observation value of 10m rather than infinity.
  • RGB supervision can recover concavities that pure mask supervision cannot, while also recovering per-voxel color.
  • Handling ray-escape events requires specifying induced observations, which may be fixed or predicted using an environment map.
  • The framework requires an event cost function differentiable with respect to ray-event probabilities and observations explainable from ray information and voxel predictions.

A2.2. Experimental Setup

The paper identifies two benchmarking biases: binary voxelization can favor 3D-supervised methods, while shared CAD models can inflate PASCAL3D performance through retrieval.

  • Biases in ‘Ground-truth’ Voxelization: Binary voxelization labels any cell containing surface as occupied, although view supervision also receives free-space evidence from rays passing through partially occupied cells.This makes the evaluation target over-inflated relative to the information available to multi-view supervision.
  • Biases in ‘Ground-truth’ Voxelization: 3D-supervised methods inherit the voxelization bias, whereas multi-view supervised methods do not, potentially explaining part of their performance gap at many views.
  • Biases in ‘Ground-truth’ Voxelization: Using IoU with binary ground truth may be inappropriate because a metric supporting soft ground truth could better reflect view-supervised evidence.
  • On Benchmarking Protocols for PASCAL 3D Dataset: Retrieving among the 65 shared PASCAL3D CAD models achieves mean IoU 0.72 across 10 categories, exceeding the reported approach.
  • On Benchmarking Protocols for PASCAL 3D Dataset: Sharing CAD models between training and testing rewards systems that bias predictions toward those models and is therefore not recommended for benchmarking.

A3. CNN Architectures, Experiment Details and Result Interpretation

Experiments use encoder-decoder CNNs trained with rendered or observed multi-view supervision, while analyses examine sampling, observation types, color reconstruction, and depth-fusion behavior.

  • Experiment Details: Object experiments assume known transformations between the input image frame and observation frames, with canonical-frame outputs for reconstruction.
  • Experiment Details: Training samples 3000 rays per instance; scene reconstruction samples 3 of 30 observations per iteration with 1000 rays per observation.
  • CNN Architectures: The basic network encodes RGB images with 2D convolutions and fully connected layers, then decodes them with 3D upconvolutions into voxel occupancies.
  • Rendered Data: ShapeNet observations are rendered from random azimuth and elevation viewpoints, with lighting variation for RGB and independent uniform pixel noise for depth.
  • ShapeNet Results: DRC is robust to observation noise, saturates with about 4–5 observations, and finds depth more informative than masks at 1–2 views.
  • Depth Fusion: Depth Fusion constructs voxel counts from ray passages and terminations, but unary fusion penalizes the true shape heavily under noisy depth observations.
  • RGB-Supervised Reconstruction: Multi-view color supervision recovers full shape and texture, but limited elevation variation produces white cloud-like artifacts below cars.
Loading 1704.06254v1…