Source-linked AI summary
iSDF: Real-Time Neural Signed Distance Fields for Robot Perception
Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, Mustafa Mukadam
TL;DR
Real-time reconstruction of non-truncated SDFs is difficult because prior room-scale approaches rely on costly voxel-grid propagation. iSDF continually trains a neural signed-distance field from posed depth images using active sampling and bounded self-supervision, achieving more accurate reconstructions and better collision-cost and gradient approximations for downstream planning.
Problem
Real-time non-truncated SDF reconstruction in room-scale environments is limited, although SDFs are important for collision avoidance and trajectory optimisation.
Method
iSDF continually optimizes a randomly initialized neural network mapping 3D coordinates to signed distance, using actively sampled posed depth observations and a bound-based self-supervised loss.
Results
iSDF produces more accurate reconstructions and better collision-cost and gradient approximations than alternatives on real and synthetic indoor datasets.
Takeaways & Limitations
The neural representation provides adaptive levels of detail, plausible filling of partially observed regions, denoising, and a more compact representation for downstream planning.
Takeaways & Limitations
Low-frequency positional embedding can oversmooth surfaces, while Eikonal regularization can oversmooth around singularities and predictions in unobserved regions lack confidence estimates.
Abstract
from arXiv · showhide
We present iSDF, a continual learning system for real-time signed distance field (SDF) reconstruction. Given a stream of posed depth images from a moving camera, it trains a randomly initialised neural network to map input 3D coordinate to approximate signed distance. The model is self-supervised by minimising a loss that bounds the predicted signed distance using the distance to the closest sampled point in a batch of query points that are actively sampled. In contrast to prior work based on voxel grids, our neural method is able to provide adaptive levels of detail with plausible filling in of partially observed regions and denoising of observations, all while having a more compact representation. In evaluations against alternative methods on real and synthetic datasets of indoor environments, we find that iSDF produces more accurate reconstructions, and better approximations of collision costs and gradients useful for downstream planners in domains from navigation to manipulation. Code and video results can be found at our project page: https://joeaortiz.github.io/iSDF/ .
I. INTRODUCTION
iSDF addresses real-time reconstruction of non-truncated SDFs with a neural field trained continually from posed depth images. It aims to overcome voxel-grid resolution limits while producing accurate, compact, and useful fields for planning.
- Motivation: Real-time non-truncated SDF reconstruction remains underexplored in room-scale environments, despite SDFs supporting collision avoidance and trajectory optimisation.SDFs are efficient for repeated collision-cost queries, whereas surface or TSDF representations are impractical for planning.
- Prior limitations: Prior real-time methods reconstruct surfaces before wavefront propagation, with voxel-grid computation limiting resolution to 5cm.This limitation applies to both CPU-based and GPU-based propagation approaches.
- Approach: iSDF uses a neural network to regress 3D coordinates to signed distances, with novel self-supervision and sampling strategies for continual learning.The system trains a randomly initialised network from a stream of posed depth images and actively replays past observations.
- Properties: Compared with voxel grids, iSDF supports adaptive detail, denoising, plausible gap filling, and a more compact representation.These properties follow from its neural-field representation.
- Results: iSDF outperforms prior methods on SDF accuracy across synthetic and real indoor datasets and improves collision-cost and gradient approximations for downstream planners.It achieves SDF error below 6cm for all sequences and is the only method reported to reconstruct watertight zero-level-set meshes.
III. SIGNED DISTANCE FIELDS
An SDF assigns each 3D point a signed distance to the closest surface, with the surface represented by the zero level set. iSDF models this function using a real-time neural network optimized from incoming measurements.
- Signed distance fields: A signed distance field maps each point in space to its signed distance from the closest surface, with negative values inside and positive values outside.The surface is represented by the field’s zero level set.
- Signed distance fields: SDF gradients encode geometric structure: they point toward the closest surface, equal the surface normal on the surface, and satisfy the Eikonal equation where defined.The Eikonal constraint is |∇_x f| = 1.
- iSDF representation: iSDF represents the signed distance function with an MLP that maps a 3D coordinate x to a signed distance value s.The model is randomly initialized and optimized online from posed depth images.
- Training: Training samples points from selected frames and minimizes losses on predicted signed distances and spatial gradients.Active frame selection supports continual learning from a sparse representative keyframe set.
- Implementation: The network uses a four-hidden-layer MLP with 256 activations per layer, softplus activations, and positional embedding of 3D coordinates.The embedding uses an off-axis construction with L = 5.
B. Active sampling
iSDF actively samples representative frames and points along back-projected rays, then uses nearby measured surface points to bound self-supervised SDF predictions. Batch-distance bounds are selected because they provide tighter supervision at little extra computation cost.
- Active sampling: Active replay selects a sparse set of representative keyframes to reduce computation and mitigate catastrophic forgetting.Subsequent keyframes are selected using an information-gain metric.
- Active sampling: Each selected frame contributes random pixels and multiple depth samples along corresponding back-projected rays, including the measured surface depth.Sampling concentrates additional points near surfaces, where the SDF is harder to model.
- Closest-surface approximation: The ideal signed-distance target is approximated using a nearby sampled surface point because computing distances to all surfaces is infeasible under partial observation.The nearby point produces a bound whose absolute value is at least the true signed distance, with equality at the closest surface point.
- Closest-surface approximation: A closer nearby surface point gives a tighter bound, creating a trade-off between computational cost and supervision quality.The evaluated choices include ray intersection, normal-corrected, and batch-distance methods.
- Closest-surface approximation: The batch-distance method provides the tightest bounds and is used in all experiments because it performs best at little extra computation cost.The figure compares bound tightness for sample points along different rays.
- Self-supervised loss: The free-space loss is zero for predictions within the positive bound, linear above it, and exponential for negative predictions.The loss is applied to points in free space.
D. Loss for self-supervised continual learning
The loss combines near-surface SDF supervision, free-space bounds, gradient alignment, and Eikonal regularisation to train the field from sparse depth-derived samples.
- SDF loss: The free-space loss is zero for positive predictions below bound b, linear above b, and exponential for negative predictions.The exponential term uses β = 5.
- SDF loss: The full SDF loss applies near-surface supervision within truncation region t and free-space loss elsewhere.Near-surface predictions are directly supervised toward the bound, while free-space predictions are constrained separately.
- Gradient loss: The gradient loss penalises cosine distance between the predicted SDF gradient and an approximated gradient from the closest surface point in the batch.For surface samples, the approximated gradient is replaced by the depth-image surface normal.
- Eikonal regularisation: Eikonal regularisation is applied only beyond a = 10cm from the ray surface intersection, improving performance and propagating the field into free space.The method avoids common gradient discontinuities near surfaces and gives regularisation a role similar to wavefront propagation.
- Optimisation: The network parameters are optimised to minimise the combined loss using Adam with specified loss weights, learning rate 0.0013, and weight decay 0.012.The reported weights are λsurf = 5, λgrad = 0.02, and λeik = 0.25.
A. Datasets
The evaluation uses synthetic ReplicaCAD sequences and real ScanNet RGB-D sequences, with 1cm voxel-grid SDFs precomputed as ground truth for each sequence.
- ReplicaCAD: ReplicaCAD provides 6 synthetic sequences generated from simulated measurements of a camera mounted on a mobile manipulator.The sequences cover two room configurations and evaluate navigation, object reconstruction, and other reconstruction properties.
- ScanNet: ScanNet contributes 3 longer and 3 shorter randomly chosen sequences captured with a handheld RGB-D camera.These sequences provide real-world indoor observations for evaluation.
- Ground truth: Ground-truth SDF values are precomputed on a voxel grid with 1cm resolution and interpolated at evaluation query points.The SDF is obtained from occupancy and inverse-occupancy Euclidean distance transforms, followed by subtraction.
B. Comparisons
iSDF is compared with voxel-based two-stage reconstruction systems, using reconstruction, collision-cost, and gradient metrics evaluated at common sampled points.
- Comparison methods: iSDF is compared against voxel-based methods that fuse depth into a surface representation before transforming it into an SDF.The second transformation stage is the dominant computational cost for both comparison methods.
- Voxblox: Voxblox uses CPU TSDF fusion and wavefront propagation from updated voxels, with 5.5cm voxels to keep SDF updates near half a second.Higher resolution would make Voxblox too slow for real-time planning.
- KinectFusion+: KinectFusion+ fuses depth into an occupancy grid and computes the SDF using a GPU-based Euclidean distance transform.The implementation uses custom CUDA kernels and is introduced because GPU methods for real-time SDF reconstruction were not known to the authors.
- Evaluation protocol: All metrics are evaluated at 200k points sampled from image pixels and depths along back-projected rays, with every method using the same points.iSDF results are averaged over 10 network initialisations, whereas Voxblox and KinectFusion+ are deterministic.
- Evaluation metrics: The metrics measure absolute SDF error, collision-cost error, and cosine distance between predicted and ground-truth SDF gradients.The collision cost follows CHOMP with ϵ = 2m and emphasizes regions near and inside surfaces.
VI. RESULTS
Across synthetic and real indoor sequences, iSDF reconstructs more complete, denoised, and plausible SDFs than voxel-grid alternatives, including partially observed regions and varying detail levels.
- Overall results: iSDF produces more accurate and complete SDFs than Voxblox and KinectFusion+ across evaluated sequences.The main paper reports results for 6 representative sequences, with remaining evidence in the supplement.
- Completeness and interpolation: iSDF predicts plausible fields in full environments, including interiors and other regions not observed by the camera.It returns predictions inside the beanbag, table, and wall, where the alternatives do not.
- Completeness and interpolation: Near visible-region edges and surfaces, iSDF provides complete reconstructions where voxel-based methods contain unmapped holes.Those holes are especially common in distant, partially occluded regions and enlarge further in the gradient field.
- Mesh quality: iSDF produces smooth, watertight zero-level-set meshes by filling gaps and denoising noisy measurements.The reported interpolation and denoising arise from network priors and self-supervised features learned from observed room regions without pretraining.
- Completeness and interpolation: A global neural representation enables accurate predictions at unseen floor levels and on the unobserved backsides of objects.Shared features across predictions support interpolation in partially observed regions.
- Adaptive detail: iSDF allocates detail adaptively, producing fine reconstructions of nearby objects while alternatives remain coarse and noisy.The paper illustrates this with a salt shaker approached for manipulation, while noting distant surface detail may be missed.
B. Quantitative results
Quantitatively, iSDF achieves the lowest SDF error across the evaluated sequences and rapidly captures room structure, while mesh extraction yields complete, denoised geometry.
- Mesh extraction: iSDF produces complete and denoised meshes from SDF queries on a uniform grid followed by marching cubes.The extracted mesh corresponds to the zero crossing of the level set.
- SDF accuracy: iSDF consistently achieves the lowest SDF error, reaching less than 6cm at the end of every sequence.It often outperforms alternatives by more than 5cm, although the gap varies by sequence.
- Planning metrics: iSDF is generally most accurate for collision costs and gradients, though KinectFusion+ performs better on gradients in some sequences.Voxblox is slightly worse on collision-cost error, while the gradient gap is smaller.
- SDF accuracy: Far from surfaces, iSDF is significantly more accurate than alternatives, whereas Voxblox can achieve similar accuracy very close to surfaces.The error analysis separates SDF accuracy by true signed distance.
- Error sources: Large voxel sizes used for real-time performance primarily account for the accuracy difference between iSDF and voxel-based alternatives.Voxblox also introduces distance errors by measuring only along horizontal, vertical, and diagonal grid lines.
- Early reconstruction: iSDF is often better across all metrics by the largest margin at the sequence start, when limited mapping makes navigation trajectories especially poor.The reconstruction rapidly captures the room’s rough structure after a few seconds.
C. Timings and memory
iSDF uses substantially less memory than voxel-grid baselines while operating in real time, though neural-field reconstruction still has detail and smoothness limitations.
- iSDF requires 1MB for network weights and around 20 keyframes, compared with 5MB for Voxblox and 30MB for KinectFusion+ voxel grids.
- A single iSDF iteration averages 33ms, including 2ms for sampling, 2ms for bounds, 10ms for the forward pass, and 19ms for backpropagation.
- With half the compute budget, iSDF uses fewer optimisation steps per second while voxel baselines increase voxel size to remain real time.
- Low-frequency positional embeddings can oversmooth surfaces, and Eikonal regularisation can oversmooth regions around singularities.
APPENDIX
The appendix evaluates supervision-bound choices and baseline behavior under constrained computation, finding that tighter batch-distance bounds improve reconstruction while exposing voxel-grid limitations.
- KINECTFUSION+: KinectFusion+ initializes voxels as unoccupied, so unseen object interiors remain unavailable for downstream SDF queries.
- Compute budget: With half the compute budget, iSDF reduces training speed from around 30 to 15 iterations per second, while voxel baselines require larger voxel sizes.
- D. SDF supervision bounds: Point batch-distance bounds produce the most accurate SDF reconstruction across all sequences, outperforming normal-correction and ray bounds.
- D. SDF supervision bounds: The average computation times for ray, normal, and batch-distance bounds are 0.2ms, 0.6ms, and 2.3ms, respectively.
C. Evaluation in Voxblox mapped region
Restricting evaluation to Voxblox’s mapped region still leaves iSDF ahead on SDF accuracy and collision-cost approximation, while its error advantage reflects accuracy both near and far from surfaces.
- Within the Voxblox mapped region, iSDF produces the most accurate SDFs and best approximations of collision costs.
- iSDF has lower SDF error than Voxblox on all sequences even after evaluation points outside the mapped region are discarded.
- iSDF and KinectFusion+ are more accurate far from surfaces, while iSDF and Voxblox are most accurate close to surfaces.
- iSDF achieves the lowest average SDF error because it maintains low error both close to and far from surfaces.
- Across additional sequences, iSDF consistently has the lowest SDF and collision-cost error, while KinectFusion+ has similarly accurate gradients.