PaperlayerView exact revision on arXiv

Source-linked AI summary

World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible

Hao Zhang, Mohamed El Banani, Jen-Hao Cheng, Paul Zhang, Yi Hua, Ben Mildenhall, Christoph Lassner, Narendra Ahuja, Gengshan Yang

arXiv:2606.13652v1cs.CVcs.GR

TL;DR

Existing image-to-3D methods struggle to combine pixel-faithful visible geometry with complete occluded structure. World Tracing addresses this with a pixel-aligned multilayer camera-space representation and achieves strong results across object, scene, and dynamic benchmarks while supporting downstream 3D applications.

  • Problem

    Existing methods separately provide pixel-faithful visible surfaces or complete canonical shapes, leaving faithful and complete camera-frame geometry unavailable.

  • Method

    World Tracing predicts ordered camera-space 3D point layers per input pixel, with WT-DiT generating visible and occluded geometry in one pixel-aligned tensor.

  • Results

    WT surpasses monocular-depth predictors on visible-surface accuracy and image-to-3D generators on Chamfer distance across object, scene, and dynamic benchmarks.

  • Takeaways & Limitations

    The representation supports text-driven 3D scene editing, geometry-conditioned novel-view video synthesis, and training-free textured-mesh generation without further per-task 3D training.

  • Takeaways & Limitations

    Most reported results use 3D-asset-only checkpoints without RGBD data, with mixed training evaluated separately for real-scene depth comparison.

Abstract

from arXiv · show

Image-to-3D methods often trade off faithfulness and completeness: depth estimators are anchored to input pixels but stop at the visible surface, while image-to-3D models generate complete shapes that are often misaligned with the input. We introduce World Tracing, a generative pixel-aligned geometry representation that predicts 3D points aligned with observed pixels while completing geometry beyond the visible surface. For each input pixel, World Tracing predicts an ordered stack of camera-space 3D points, where the first layer represents the visible surface and subsequent layers represent front-to-back intersections with occluded surfaces. We instantiate this representation with a world-tracing diffusion transformer, WT-DiT, which treats multiple geometry layers as separate denoising tokens coupled through factorized and global attention. WT-DiT is trained with pixel-space flow matching and a mixed noise schedule that balances visible-surface reconstruction with occluded-geometry generation. World Tracing achieves strong performance on visible-surface reconstruction and complete geometry generation across object, scene, and dynamic benchmarks, outperforming both depth predictors and image-to-3D generators. It also preserves 2D-to-3D correspondence, enabling text-driven 3D scene editing, geometry-conditioned novel-view video synthesis, and training-free integration with textured-mesh generators.

1 Introduction

World Tracing introduces a pixel-aligned multilayer camera-space geometry representation that unifies faithful visible-surface reconstruction with occluded-geometry completion. WT-DiT instantiates it for scalable generation and downstream 3D applications across objects, scenes, and videos.

  • Motivation: Existing methods either reconstruct visible pixels without completing geometry or generate complete canonical shapes without pixel alignment.Neither paradigm provides faithful and complete geometry in the camera frame.
  • World Tracing: World Tracing predicts an ordered stack of L camera-space 3D points per input pixel, with visible surfaces first and occluded geometry in deeper layers.Visible reconstruction and generative completion become successive layers of one tensor on the input pixel grid.
  • WT-DiT: WT-DiT is a flow-matching diffusion transformer that models geometry layers as interacting denoising tokens with efficient three-way factorized attention.Its depth-filling objective simplifies prediction heads, while image-grid layers inherit priors from pretrained image encoders.
  • Applications: The representation supports text-driven 3D scene editing, geometry-conditioned novel-view video synthesis, and training-free textured-mesh generation.These downstream uses are demonstrated without further per-task 3D training.
  • Evaluation: Across object, scene, and dynamic benchmarks, WT improves visible-surface accuracy and complete geometry, surpassing monocular-depth predictors and image-to-3D generators on their respective evaluations.The evaluation measures both plausibility and geometric consistency, including Chamfer distance to ground-truth geometry.

2 Related Work

Prior 3D reconstruction methods evolved from monocular depth and pointmaps to multi-view reconstructors, but single-point-per-pixel representations omit occluded geometry. Image-to-3D and 4D methods generate complete or dynamic geometry through canonical, structured, or optimized representations, whereas WT retains camera-aligned pixel correspondence.

  • Monocular and multi-view 3D reconstruction: Pixel-aligned reconstruction methods progressed from monocular depth and depth diffusion to implicit-intrinsics pointmaps and multi-view reconstructors, but retain one 3D point per pixel.This structural limit leaves geometry behind the first visible surface absent; layered depth images and neural variants relax it.
  • Image-to-3D generation: Image-to-3D pipelines use feed-forward generation, multi-view reconstruction, or SDS optimization, producing complete geometry but typically losing input pixel alignment in a canonical object frame.These methods span radiance-field latents and structured-latent or Gaussian spaces, while WT keeps the camera-aligned pixel grid as its native coordinate system.
  • Video-to-4D and feed-forward 4D: Video-to-4D methods rely on sequence optimization or faster feed-forward predictors, which commonly represent dynamic geometry as single-surface point clouds, canonical meshes or Gaussians, spacetime latents, or trajectory fields.Optimization-based approaches may also use auxiliary trackers or part templates; the passage contrasts these representations with WT-D.

3 Method

World Tracing represents geometry as an ordered, pixel-aligned stack of camera-space 3D points, with visible surfaces in layer 0 and hidden-surface completion in deeper layers. WT-DiT generates this stack through flow matching, multilayer attention, dense targets, and validity-aware training without per-layer masks.

  • WT-DiT: WT-DiT conditions a flow-matching diffusion transformer on frozen MoGe pixel-level features and denoises a multilayer XYZ tensor directly on the input image grid.The model predicts the clean normalized coordinate endpoint and integrates the flow ODE from noise toward t=0 using 20 ODE steps.
  • Representation: World Tracing predicts an ordered front-to-back stack of camera-space 3D points for each input pixel, preserving pixel-to-3D correspondences and camera pose.Layer 0 is the visible surface, while deeper layers represent hidden surfaces and increasingly conditional generation.
  • Representation: Dense targets fill missing deeper-layer intersections by repeating the nearest earlier valid point, avoiding per-layer validity masks without changing the recovered shape.Overlapping points allow terminated rays to collapse onto the front surface and reduce severe deeper-layer mask imbalance.
  • Training: Training masks invalid pixels in the loss and replaces their noisy geometry with maximum Gaussian noise, while a soft adjacent-layer penalty encourages front-to-back ordering.The method uses no per-layer mask, silhouette, or visibility classification head.
  • WT-DiT: Three-way attention alternates within-layer image attention, along-ray layer attention, and global attention to model spatial structure, depth ordering, and object- or scene-level context.Layer FiLM identifies the represented layer, while shared AdaLN provides diffusion-time conditioning.

4 Experiments

Experiments test whether pixel-aligned multilayer geometry improves faithful visible reconstruction and complete geometry across objects, scenes, and dynamic clips, while supporting downstream geometry-dependent pipelines. Results show strong performance across these settings, alongside ablations of training and representation choices.

  • Evaluation: Evaluation emphasizes visible-surface depth errors and complete-geometry Chamfer/F-score, with visible and occluded breakdowns where applicable.Generative methods use K=8 random samples, and scene editing and view synthesis are presented qualitatively.
  • Objects: WT-O achieves the best visible-surface accuracy and complete-object geometry, showing that occluded-geometry generation does not reduce visible-surface faithfulness.This holds across MAE, RMSE, and AbsRel for visible surfaces, while complete geometry improves over textured-mesh baselines.
  • Scenes: WT-S achieves the best visible-surface depth and L0 point-cloud geometry on held-out scenes, while substantially improving all-layer geometry over LaRI-scene.Evaluation includes the held-out 3D-FRONT benchmark, a 200-sample internal probe, and out-of-distribution generated room images.
  • Dynamic clips: WT-D wins on Obj.-Val and Truebone and achieves the best mean CD-L2, while ActionMesh is strongest on ActionBench.The three splits target held-out animated objects, explicitly articulated assets, and action-driven motions, respectively.
  • Design ablations: Joint depth-and-mask training is unstable and imbalanced, with gradient cosine similarity reaching −0.19 at 500 iterations and valid area falling from 8.14% at L0 to 0.60% at L5.These observations motivate depth filling rather than direct mask prediction.
  • Design ablations: WT uses a layer-aware timestep mixture that favors faithful visible-layer prediction while treating occluded layers more like conditional generation.The schedule extends pixel-aligned geometry modeling from single-layer regression and diffusion to multilayer flow matching.

5 Downstream Pipeline Demonstrations

World Tracing transfers its complete, camera-space, pixel-aligned geometry to downstream pipelines for object insertion, textured mesh generation, and geometry-guided video synthesis. These demonstrations preserve 2D-to-3D correspondence while addressing incomplete visible-layer conditioning and pose misalignment.

  • Shared representation benefits: Across downstream demonstrations, WT’s complete input-camera geometry preserves intrinsics and pixel-level correspondence for object insertion, textured mesh generation, and geometry-guided video synthesis.Figure 3 summarizes this shared pattern across the three applications.
  • Object insertion: WT enables object insertion by lifting 2D-edited images into 3D while preserving image-grid correspondence.The pipeline predicts the original scene with WT-S and the generated object or edited region with WT-O before insertion.
  • Geometry-guided video synthesis: WT provides complete geometry memory for video synthesis, preventing newly exposed regions from being invented from scratch after camera motion.Its multilayer geometry is pixel-aligned with the input image and integrates with image-to-video conditioning pipelines.
  • Pose-aligned mesh generation: WT supports pose-aligned textured mesh generation by supplying TRELLIS with camera-space, pixel-aligned sparse structure instead of canonical-frame geometry.The WT point stack is voxelized for later TRELLIS stages, retaining recoverable input-camera information and improving reprojection consistency.

6 Conclusion

World Tracing presents pixel-aligned multilayer geometry that unifies visible-surface fidelity and occluded-geometry completion in a camera-space tensor. WT-DiT learns this representation across objects, scenes, and dynamic clips using a frozen 2D foundation encoder, flow matching, and an XYZ-only depth-filling objective.

  • 6 Conclusion: World Tracing represents visible surfaces and occluded geometry as successive layers of one pixel-aligned camera-space tensor.As layer depth increases, occluded structures behind near surfaces are filled while Layer 0 remains aligned with the input.
  • 6 Conclusion: WT-DiT learns World Tracing with a frozen 2D foundation encoder, flow matching, and an XYZ-only depth-filling objective.The approach applies across objects, scenes, and dynamic clips.
  • 6 Conclusion: The multilayer predictions fill occluded geometry such as floors, walls, room interiors, and structures behind furniture while preserving pixel alignment at Layer 0.This behavior appears on held-out 3D-FRONT frames and out-of-distribution generated indoor rooms.

Technical Appendices and Supplemental Material

The supplementary material provides deferred methodological, training, evaluation, statistical, qualitative, and discussion details for World Tracing. It also documents limitations and future work.

  • Method details: Appendix A details scale normalization, invalid-pixel noise filling, and the monotonicity penalty.
  • Training and architecture: Appendix B covers the training schedule, optimizer, model hyperparameters, decoder, and layer-aware diffusion-time schedule.
  • Evaluation and statistics: Appendix C specifies the rendering pipeline, source datasets, evaluation splits, baselines, and metric definitions, while Appendix D reports per-layer validity statistics on the object corpus.
  • Analysis and discussion: Appendices E and F provide qualitative explanations for WT-O and WT-S, plus discussion of layer count, prediction target, architecture, generalization, limitations, and future work.

A Method Details

The method normalizes camera-space geometry differently for object/dynamic-object data and scenes while retaining reversible metric coordinates. It also handles invalid pixels with Gaussian noise and encourages front-to-back layer ordering through a soft monotonicity penalty.

  • Scale normalization: Object and dynamic-object data use global per-channel z-score normalization, while scenes use a per-sample log-median coordinate map.The scene map uses the median valid depth over the multilayer stack and logarithmic transforms for depth and lateral coordinates.
  • Scale normalization: Both coordinate maps are reversible, and the same architecture predicts normalized tensors in both regimes; only the input/output coordinate transform changes.Predictions can therefore be returned to camera-space metric coordinates at inference.
  • Invalid input pixels and noise fill: Invalid pixels are excluded from endpoint loss and replaced with fresh Gaussian noise across all geometry layers and XYZ channels during training and inference.Layer-0 alpha validity is broadcast across layers and channels before patchification.
  • Invalid input pixels and noise fill: Invalid-pixel corruption makes geometry tokens uninformative there, while valid rays receive targets for every layer without an auxiliary per-layer mask head.Dense forward fill defines predictions along valid input rays, with layer-0 alpha as the only visibility input.
  • Monotonicity penalty: A one-sided adjacent-layer monotonicity penalty enforces normalized front-to-back ordering, with λmono=0.1 in the full training loss.The penalty is zero when ordering is correct and stabilizes early training without measurably affecting final geometry quality.

B Training Details

Training combines a staged high-resolution recipe with frozen image encoding, temporal adaptation, and layer-aware diffusion schedules. Additional procedures improve robustness to imperfect silhouettes and recover camera intrinsics directly from predicted geometry.

  • Backbone and resolution: Object and dynamic-object models use 504×504 inputs, patch size 14, six geometry layers, and a frozen MoGe ViT-L encoder whose projected features feed the decoder.The object backbone uses 48 pre-norm DiT blocks at width D=1536 with 24 attention heads; only the encoder-to-decoder projection is updated.
  • Temporal attention block: For 16-frame clips, temporal attention operates along time after each global-attention block, using 1D RoPE, shared time AdaLN, and LayerScale initialized at γ=10−5.Warm-starting from a single-frame WT-O checkpoint is designed to reproduce static behavior initially.
  • Training schedule: Training uses 100K iterations at 196×196 followed by 100K at 504×504, while the dynamic model is initialized from the static checkpoint and fine-tuned for 50K clip iterations.The dynamic model adds temporal attention blocks before clip fine-tuning.
  • Robust silhouettes: Training jitters alpha boundaries and assigns newly exposed or missing-support pixels filled geometry targets at reduced weight, improving robustness to mask errors without a separate silhouette loss.The procedure addresses shifted boundaries, missed thin parts, and small holes within the visible support.
  • Self-consistent intrinsics: Unknown camera intrinsics are recovered by least-squares fitting projection equations to valid layer-0 camera-space points, yielding parameters self-consistent with the pointmap and pixel grid.The fitted parameters are (fx, fy, cx, cy), supporting camera-aware uses such as object insertion and novel-view video synthesis.
  • Layer-aware diffusion-time schedule: The layer-aware curriculum mixes standard and plateaued logit-normal timestep distributions, with the former emphasizing t≈0.5 for deeper layers and the latter allocating more probability to small t.Figure 6 describes their equal-weight 50/50 mixture in phase (ii).

C Data Pipeline Details … E.2 Why WT-S preserves planar structure

The appendix specifies World Tracing’s data, multilayer supervision, evaluation protocol, and validity statistics, then explains qualitative gains from diffusion and pixel alignment. It also reports why WT-S preserves planar structure on out-of-distribution images.

  • C Data Pipeline Details: Object data spans roughly 300K unique assets and 17M rendered views, while scenes use 3D-FRONT and an internal curated corpus, and dynamic data includes animated assets.Rendered inputs use randomized lighting, viewpoints, and camera intrinsics.
  • C Data Pipeline Details: WT-S mixes multilayer 3D-asset data with a 12-dataset RGBD-style corpus that supervises only the visible layer L0.For RGBD-style frames, bsingle=1, so deeper predictions L1 through L5 are shaped exclusively by the multilayer corpus.
  • C Data Pipeline Details: Evaluation samples held-out object assets, uses a held-out 3D-FRONT scene split, separately reports an internal test set, and evaluates dynamic assets with Obj.-Val, Truebone, and ActionBench.Baselines include depth predictors, layered predictors, image-to-3D generators, dynamic-geometry methods, and TRELLIS hybrids.
  • C Data Pipeline Details: Depth peeling generates ordered multilayer ground truth by repeatedly rendering from one camera and exposing the next intersection after ignoring previously captured surfaces.The resulting layers preserve the true ordered set of ray intersections, with target filling handling rays having fewer than L intersections.
  • D Layer Validity Statistics: Mean valid pixel coverage falls from 8.14% at L0 to 0.60% at L5, creating a more than 13× imbalance that causes per-layer mask prediction to collapse on deeper layers.Coverage is measured over the full 504×504 rendered image, including background outside the object silhouette.
  • E Geometry Generation: Qualitative Analysis: The qualitative analysis attributes object gains to diffusion’s modeling of multimodal occluded geometry and pixel alignment’s preservation of image correspondence during complete-shape prediction.LaRI’s regression-style head tends to average invisible back-side geometry, while TRELLIS-style canonical-frame generators can produce plausible but input-unfaithful meshes.
  • E.1 Why WT-O outperforms regression-based and canonical-frame baselines: A reduced-layer WT-O control with a LaRI-comparable data budget shows the gains are not solely attributable to scaling the dataset.This supports an architectural contribution beyond increased training-data scale.
  • E.2 Why WT-S preserves planar structure: WT-S is more complete and stable than LaRI-scene and can produce faithful geometry on some out-of-training-set indoor and outdoor images, where MoGe-2 bends planar or vertical structures.The examples include facades and walls that should remain planar or vertical.

E.3 Real-scene depth benchmarks … F.2 XYZ pointmaps versus depth plus intrinsics

WT-S remains competitive on real-scene visible-surface depth while mix-training substantially improves benchmark performance, and the discussion motivates six layers and direct XYZ prediction as practical design choices. The method prioritizes multilayer completion beyond visible surfaces rather than surpassing dedicated depth predictors alone.

  • E.3 Real-scene depth benchmarks: WT-S is evaluated on NYU Depth V2 and ETH3D using SSI-aligned visible-surface depth metrics at evaluation resolution.NYU contains 1,449 indoor frames, while ETH3D contains 454 frames across seven indoor and six outdoor scenes.
  • E.3 Real-scene depth benchmarks: Mix-training reduces WT-S AbsRel by 4.0% on NYU, 13.3% on ETH3D-indoor, and 7.1% on ETH3D-outdoor under 20-step inference.With 50 denoising steps, the gains widen to 6.0%, 16.6%, and 15.4%, respectively.
  • E.3 Real-scene depth benchmarks: The 50-step mix-trained WT-S achieves the best ETH3D-indoor AbsRel and NYU δ<1.25, while ranking second to Pi3X on NYU and ETH3D-outdoor AbsRel.Its reported values are 0.0332 on ETH3D-indoor, 0.0374 on NYU, 0.9871 for NYU δ<1.25, and 0.0451 on ETH3D-outdoor.
  • E.3 Real-scene depth benchmarks: WT-S’s central objective is to add occluded back-layer geometry while remaining competitive on visible surfaces, rather than surpassing dedicated single-layer depth predictors.The 3D-asset-only checkpoint reaches indoor L0 parity using a much smaller multilayer-only training corpus.
  • F Additional Discussion: The appendix frames its additional experiments as expansions of design choices summarized in the main paper, while retaining depth-filling and timestep-sampling ablations there.This establishes the scope of the additional discussion rather than reporting a new benchmark result.
  • F.1 How many layers?: Six layers are a practical sweet spot because valid support drops rapidly after the first two surfaces, although later layers capture thin structures and repeated occlusions.More layers could help rare perforated shapes but increase memory use and make sparse-tail supervision harder.
  • F.2 XYZ pointmaps versus depth plus intrinsics: Direct camera-space XYZ prediction avoids the global shape warping that can result when depth prediction relies on erroneous focal-length or principal-point estimates.The passage argues that XYZ produces more plausible global shapes, especially when crop, mask, and camera metadata are uncertain.

F.3 Architecture details · F.4 Generalization beyond the training regime · F.5 Limitations and future work

The final architecture choices improve optimization and support spatial extrapolation, while World Tracing generalizes beyond object training to scenes and multiple objects. Its remaining constraints include fixed layer capacity, rendered supervision, iterative sampling, synthetic-to-real artifacts, and limited long-range video modeling.

  • F.3 Architecture details: LayerScale initialization of 10−4 reduced total loss from 0.009258 to 0.007018 at 10k iterations.In the same ablation, XYZ loss decreased from 0.007242 to 0.003537.
  • F.3 Architecture details: RoPE was neutral in the small ablation but remains useful for full-model extrapolation across resolutions and camera crops.Its role is tied to spatial and ray-wise attention in the full model.
  • F.4 Generalization beyond the training regime: The object model produces reasonable geometry on scene images despite receiving no scene training.This behavior suggests that pixel-aligned multilayer prediction transfers across object and scene regimes.
  • F.4 Generalization beyond the training regime: The object model is robust to multiple objects in one image despite not being explicitly trained with them.The passage contrasts this behavior with canonical image-to-3D generation under cluttered-scene conditions.
  • F.5 Limitations and future work: World Tracing is bounded by a fixed layer count and rendered supervision.Highly perforated geometry may require more than six surfaces.
  • F.5 Limitations and future work: Synthetic-to-real artifacts remain on textureless or reflective regions.This limitation is identified alongside the constraints of fixed layers and rendered supervision.
  • F.5 Limitations and future work: Real-time applications require distillation or fewer sampling steps because WT still relies on iterative flow sampling.Reducing sampling cost is presented as a future requirement for deployment.
  • F.5 Limitations and future work: The dynamic model operates on short clips, leaving long-range memory and persistent identities across long videos for future work.The passage identifies extending WT-D as a natural direction.
Loading 2606.13652v1…