PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiniDepth: Arbitrary-Resolution and Fine-Grained Depth Estimation with Neural Implicit Fields

Hao Yu, Haotong Lin, Jiawei Wang, Jiaxin Li, Yida Wang, Xueyang Zhang, Yue Wang, Xiaowei Zhou, Ruizhen Hu, Sida Peng

arXiv:2601.03252v1cs.CV

TL;DR

Existing grid-based depth estimators are limited in output resolution and fine-detail recovery. InfiniDepth replaces this representation with a neural implicit field and continuous local decoding, achieving strong results across relative and metric depth benchmarks while also improving novel-view synthesis under large viewpoint shifts.

  • Problem

    Grid-based depth prediction is constrained to fixed image locations, limiting arbitrary-resolution output and fine-detail recovery, while real-world evaluation depth is often sparse and low-resolution.

  • Method

    InfiniDepth represents depth as a neural implicit field queried at continuous 2D coordinates, and combines it with a depth-query strategy that allocates sub-pixel samples using surface geometry.

  • Results

    InfiniDepth achieves state-of-the-art performance across relative and metric depth estimation on synthetic and real-world benchmarks, particularly in fine-detail regions, and benefits novel-view synthesis under large viewpoint shifts.

  • Takeaways & Limitations

    The representation supports arbitrary-resolution, fine-grained depth estimation and produces more useful surface points for novel-view synthesis with fewer holes and artifacts.

  • Takeaways & Limitations

    The model is trained only on single-view depth data and does not explicitly enforce temporal consistency for videos, so frames may flicker.

Abstract

from arXiv · show

Existing depth estimation methods are fundamentally limited to predicting depth on discrete image grids. Such representations restrict their scalability to arbitrary output resolutions and hinder the geometric detail recovery. This paper introduces InfiniDepth, which represents depth as neural implicit fields. Through a simple yet effective local implicit decoder, we can query depth at continuous 2D coordinates, enabling arbitrary-resolution and fine-grained depth estimation. To better assess our method's capabilities, we curate a high-quality 4K synthetic benchmark from five different games, spanning diverse scenes with rich geometric and appearance details. Extensive experiments demonstrate that InfiniDepth achieves state-of-the-art performance on both synthetic and real-world benchmarks across relative and metric depth estimation tasks, particularly excelling in fine-detail regions. It also benefits the task of novel view synthesis under large viewpoint shifts, producing high-quality results with fewer holes and artifacts.

1. Introduction

InfiniDepth addresses the fixed-resolution and detail limitations of grid-based depth estimation by representing depth as neural implicit fields with continuous local queries. It introduces Synth4K and reports strong performance across depth-estimation benchmarks, fine-detail regions, and novel-view synthesis under large viewpoint shifts.

  • Motivation: Grid-based depth representations constrain output resolution and struggle to preserve fine details in regions with substantial geometric variation.These limitations arise because depth is predicted only at fixed grid locations, often requiring upsampling or projection for larger outputs.
  • Method: InfiniDepth predicts depth at continuous 2D coordinates using a localized neural implicit representation, enabling arbitrary-resolution maps and finer geometric detail.The method gathers multi-scale spatial features within a local window and uses a lightweight MLP to predict depth.
  • Novel-view synthesis: Pixel-aligned depth can create density imbalance in 3D point clouds, degrading novel-view synthesis under large viewpoint shifts.The imbalance is attributed to perspective projection and surface orientation.
  • Novel-view synthesis: A sub-pixel query strategy allocates query budgets according to each pixel’s corresponding 3D surface element, producing more spatially uniform points and fewer novel-view artifacts.The paper connects uniform 3D points with higher-quality novel-view synthesis under large viewpoint shifts.
  • Evaluation: Synth4K provides 4K ground-truth depth from five games, while experiments report state-of-the-art results across relative and metric depth estimation, especially in fine-detail regions.The benchmark covers diverse scenes with rich geometric and appearance details.
  • Contributions: The paper combines neural implicit depth fields, uniform-surface point queries, and Synth4K as its main contributions.These contributions target arbitrary-resolution estimation, novel-view synthesis, and high-resolution fine-detail evaluation.

2. Related Work

The related work spans relative and metric depth estimation, neural implicit representations, and the paper’s feature-query architecture. Existing depth methods commonly use ViT-based grid decoders or additional geometric inputs, whereas InfiniDepth builds on continuous implicit-function ideas.

  • Relative depth estimation: Relative-depth methods commonly use ViT backbones with convolutional decoders to regress discretized depth maps, alongside diffusion-based alternatives.The cited examples include methods that improve generalization or geometric accuracy through additional data or supervision.
  • Metric depth estimation: Metric-depth methods address scale ambiguity with camera intrinsics, sparse-depth inputs, iterative scale-shift optimization, or depth-prompt modules.These approaches target metric accuracy using different forms of geometric or auxiliary information.
  • InfiniDepth overview: InfiniDepth’s overview combines continuous feature querying from multi-layer ViT features with hierarchical depth decoding through a lightweight MLP head.The architecture is organized around feature query and depth decoding stages.
  • Implicit neural representations: Neural implicit representations map continuous coordinates to signals and have been applied to radiance fields, human geometry, images, optical flow, and multi-view scenes.InfiniDepth extends this continuous-coordinate paradigm to depth estimation.

3. Method

InfiniDepth models depth as a continuous neural implicit field and decodes each query from localized multi-scale image features. Its depth-query strategy uses differentiable geometry to allocate sub-pixel samples for approximately uniform surface coverage.

  • Implicit depth representation: The method estimates depth for any continuous 2D image-plane coordinate from a single RGB image.This formulation treats depth as a function queried at arbitrary coordinates rather than as a fixed grid output.
  • Implicit depth representation: Neural implicit fields represent signals as coordinate-conditioned neural functions whose fidelity is not tied to a fixed discretization.InfiniDepth applies this representation to depth conditioned on the input image.
  • Local implicit decoder: A multi-scale local implicit decoder reassembles encoder features, queries local descriptors at each scale, hierarchically fuses them, and predicts depth with an MLP.The decoder combines high-resolution detail with deeper semantic features.
  • Feature query: Continuous query coordinates are mapped to each feature-pyramid scale, where local neighborhoods are aggregated through bilinear interpolation.The resulting scale-specific feature tokens are used for subsequent depth decoding.
  • Infinite depth query: The depth-query strategy uses implicit-field geometry to counteract point-cloud density imbalance caused by perspective projection and surface orientation.It estimates differential surface area from depth and surface-normal/viewing-direction factors.
  • Infinite depth query: Adaptive query budgets distribute sub-pixel coordinates within each pixel patch, yielding approximately uniform surface coverage after back-projection.The queried continuous depths are back-projected to form the point cloud.
  • Implementation: The implementation uses a DINOv3 ViT-Large encoder, synthetic training data, and sparse coordinate-depth supervision with an L1 loss.The sparse supervision samples coordinate-depth pairs instead of supervising the entire depth map.

4. Experiments

InfiniDepth is evaluated on Synth4K and real-world benchmarks for relative and metric depth estimation, with targeted tests of high-resolution and fine-detail prediction. The experiments also analyze its representation and decoder, and assess its use for novel view synthesis under large viewpoint shifts.

  • 4.1. Synth4K: Synth4K provides 4K RGB-D data from five games and high-frequency masks for targeted evaluation of fine-detail depth prediction.Its higher resolution and detail coverage are intended to support high-resolution and fine-grained evaluation.
  • 4.2. Experimental Setup: Relative-depth experiments evaluate zero-shot performance on Synth4K at full 4K resolution and in high-frequency masked regions, alongside five real-world datasets.The reported metrics include δ0.5, δ1, and δ2 on Synth4K and δ1 on real-world benchmarks.
  • 4.2. Experimental Setup: Metric-depth experiments use sparse depth inputs and evaluate δ0.01, δ0.02, and δ0.04 on Synth4K and real-world benchmarks.These stricter thresholds reflect the higher accuracy typically obtained with sparse depth inputs.
  • 4.3. Comparisons with the State of the Art: On Synth4K, InfiniDepth significantly outperforms existing methods across all reported metrics, while on real-world data its relative model is on par with current SOTA and its metric model improves over existing methods.Baselines are upsampled to 4K, whereas InfiniDepth is queried directly at 4K.
  • 4.4. Ablations and Analysis: Ablations show that neural implicit fields substantially improve metric-depth estimation and moderately improve relative-depth estimation over a discrete-grid baseline, while recovering finer geometric details.Multi-scale feature querying also brings significant improvements across datasets.
  • 4.5. Application: Single-View Novel View Synthesis: InfiniDepth combined with its depth-query strategy improves single-view novel view synthesis under large viewpoint shifts.The strategy generates uniformly distributed surface points, addressing holes and artifacts associated with pixel-aligned depth.

5. Conclusion and Discussions

InfiniDepth represents depth with neural implicit fields to support arbitrary continuous queries and preserve fine-grained geometry. Its depth-query strategy also benefits novel view synthesis, while its monocular single-view training leaves temporal consistency in videos as an open limitation.

  • Conclusion: Neural implicit fields enable depth estimation at arbitrary continuous 2D coordinates while better preserving fine-grained geometric details.The formulation is validated on relative and metric depth benchmarks.
  • Conclusion: The depth-query strategy further benefits single-view novel view synthesis under large viewpoint shifts.The paper presents this as an application of the proposed depth representation and query strategy.
  • Limitations and Future Work: Because the model is trained only on single-view depth data, video applications do not explicitly enforce temporal consistency and may exhibit flickering across frames.Future work proposes extending the representation to multi-view settings for temporal stability and 3D consistency.

Supplementary Material

The supplementary material describes adaptive sub-pixel querying from normalized pixel weights and the point-wise Gaussian-splatting head built from the resulting uniform 3D points.

  • Infinite Depth Query: Adaptive querying normalizes pixel weights into a probability distribution before constructing a cumulative distribution function for sampling.The CDF maps each pixel index to the cumulative probability mass through that index.
  • Infinite Depth Query: Uniformly stratified inverse-transform sampling selects pixel indices whose frequencies match the normalized distribution.The procedure generates uniformly spaced target values and finds the smallest corresponding pixel indices.
  • Infinite Depth Query: Selected pixels receive random sub-pixel jitter within [−0.5, 0.5] around their centers before coordinate normalization.The normalized coordinates follow the model’s coordinate convention.
  • Gaussian-Splatting Head: The Gaussian-splatting head combines per-point color and Pl¨ucker ray features with ViT features to predict Gaussian attributes through MLP and linear heads.Predicted attributes include position, color, scale, opacity, and rotation terms.

A.4. Training Strategies

The training strategy normalizes depth in logarithmic space, supervises continuous-coordinate predictions on original-resolution depth, and trains the Gaussian head with reconstruction and perceptual losses.

  • Depth Normalization: Depth normalization first converts ground-truth depths to logarithmic space to reduce variance between scenes.The normalized depth uses the 2% and 98% logarithmic-depth quantiles as bounds.
  • Training InfiniDepth: InfiniDepth trains on randomly sampled coordinate-depth pairs from the original-resolution ground-truth map, using 100k pairs per image for L1 supervision.The RGB image is resized while the ground-truth depth retains its original resolution.
  • Training GS Head: The Gaussian-splatting head initializes from pretrained InfiniDepth weights, freezes the ViT encoder, and combines L1 reconstruction with LPIPS supervision.The losses target both low-frequency color reproduction and high-frequency structural fidelity.
  • Efficiency Analysis: The decoder has the lowest parameter count among compared methods, but runs slower than DepthAnythingV2 and MoGe-2.The supplementary comparison links the faster convolutional decoders to weaker fine-grained detail capture.

B. Dataset Details

The supplementary dataset material details Synth4K’s 4K game-derived scenes and the high-frequency-mask pipeline used to target fine geometric structures.

  • Dataset Curation: Synth4K contains diverse indoor and outdoor scenes from five games, collected with 3840x2160 RGB images and corresponding depth maps.The scenes use high-quality graphics and realistic lighting effects.
  • High-Frequency Mask: High-frequency structures are identified with a geometric energy map emphasizing local curvature and fine-scale depth variations.The pipeline applies multi-scale filtering, 4-neighborhood Laplacian responses, and per-pixel maximum aggregation.
  • High-Frequency Mask: The geometric energy is normalized using its 98th percentile to suppress extreme outliers.The normalization uses q0.98(E) as the reference quantile.
  • Candidate Sampling: Temperature sharpening controls high-frequency contrast: τ < 1 emphasizes sharp structures, whereas τ > 1 produces a flatter distribution.Sampling probabilities are then formed and used to draw high-frequency candidate locations with multinomial sampling.
  • Training Data: Additional training data include MatrixCity, MVS-Synth, Blendedmvs, CREStereo, FSD, and DynamicReplica.

C.1. Evaluation Protocols

The evaluation protocols standardize input resolutions and task-specific evaluation, while the NVS setup tests unseen scenes using uniform point clouds rendered through a Gaussian-splatting head.

  • Evaluation Protocols: All methods use consistent input resolutions and evaluation protocols for fair comparison.
  • Resolution Handling: On Synth4K, baselines are bilinearly upsampled to 4K, whereas InfiniDepth is queried directly at 4K.Real-world inputs use 504 × 672, while Synth4K inputs use 504 × 896.
  • Ablations: Metric-depth, relative-depth, and design-choice ablations are reported in Tables 7–9.The table captions identify the corresponding supervision, representation, and design-choice comparisons.
  • Depth Evaluation: Relative-depth evaluation applies scale-and-shift alignment, while metric-depth evaluation uses 1500 sparse ground-truth points without alignment.
  • Novel View Synthesis: For single-view NVS, the Gaussian-splatting head renders unseen scenes from an input RGB image and uniformly distributed point clouds, producing fewer artifacts and holes.The setup is trained on a Waymo training subset and evaluated on unseen scenes.

C.3. More Ablation Studies

The ablations examine supervision, depth representation, feature-query design, and benchmark examples. Results favor sub-pixel supervision and feature-pyramid attention, while neural implicit fields mainly improve visual quality for relative depth.

  • Supervision strategies: Sub-pixel supervision improves depth prediction accuracy and fine-grained prediction by supervising continuous coordinates with higher-resolution ground truth.It is compared with pixel-wise supervision, which uses ground truth downsampled to the input resolution.
  • Depth representation: Neural implicit fields noticeably enhance relative-depth visual quality, although their metric accuracy does not improve significantly.
  • Synth4K: Synth4K examples contain RGB images, depth maps, and high-frequency masks from five games: CyberPunk 2077, Marvel’s Spider-Man 2, Miles Morales, Dead Island, and Watch Dogs.
  • Design choices of implicit decoder: Attention on feature pyramids achieves the best performance with the least computational cost among the evaluated implicit-decoder designs.Alternative designs add parameters and computation without improving performance.

D. More Results

Additional results compare InfiniDepth with existing methods for relative depth estimation and single-view novel view synthesis. The comparisons cover point clouds, depth maps, RGB scenes, and large-viewpoint-shift synthesis outputs.

  • Relative depth estimation: Relative-depth point-cloud comparisons include InfiniDepth, MoGe, MoGe-2, and PPD, with InfiniDepth demonstrating strong fine-grained depth-estimation capability.
  • Synth4K: Synth4K RGB examples span five games: CyberPunk 2077, Marvel’s Spider-Man 2, Miles Morales, Dead Island, and Watch Dogs.
  • Relative depth estimation: Additional relative-depth map comparisons include Marigold, DepthAnythingV2, PPD, and MoGe-2 as baseline methods.
  • Single-view novel view synthesis: Single-view novel-view-synthesis comparisons evaluate InfiniDepth against ADGaussian under large viewpoint shifts.
Loading 2601.03252v1…