PaperlayerView exact revision on arXiv

Source-linked AI summary

FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute

Hongchi Xia, Tianhang Cheng, Wei-Chiu Ma, Shenlong Wang

arXiv:2609.08848v2cs.CVcs.RO

TL;DR

Cluttered, partially observed indoor scenes require systems to parse instances, complete geometry, synthesize appearance, and preserve layout, yet existing approaches often address only parts of this task. FIRE3D reconstructs object-level textured scenes through feed-forward perception and compact, batched object generation. From a 60-frame RGB video, it reconstructs up to twelve textured instances in under 60 seconds, including preprocessing.

  • Problem

    Fast feed-forward reconstruction of complete textured object-level scenes from unsegmented, posed RGB-D observations estimated from raw RGB captures remains a gap.

    Source: §1 · p. 2

  • Method

    FIRE3D predicts object validity, poses, and instance masks from posed RGB-D observations, then uses point-cloud-conditioned cascaded flow matching to generate object structure, shape, and material latents.

    Source: §1 · p. 2

  • Results

    From a 60-frame RGB video, FIRE3D reconstructs up to twelve textured instances in under 60 seconds, including Pi3X preprocessing, and leads mAP on all three evaluated datasets.

    Source: §1 · p. 2, §4.3 · p. 8

  • Takeaways & Limitations

    FIRE3D produces independently editable textured scene assets for rendering and interactive applications.

    Source: §1 · p. 2, §5 · p. 9

    FIRE3D focuses on static indoor scenes; physical stability and sim-to-real transfer are not systematically evaluated.

    Source: §5 · p. 9, §4.4 · p. 9

Abstract

from arXiv · show

We present FIRE3D, a unified framework that transforms a single RGB image or casual video into interactable 3D scene assets for games and interactive applications in under a minute for up to twelve textured instances including preprocessing. At the core of FIRE3D is a feed-forward inference pipeline that predicts a compositional scene representation from posed RGB-D observations estimated from the RGB capture, including the 6-DoF pose, bounding box, mesh, and texture for detected objects. By modeling the scene as a collection of discrete entities, FIRE3D produces amodal object assets that can be independently edited and used in interactive applications. Our framework requires no test-time optimization, runs substantially faster than the compared reconstruction systems, and provides object-level completeness beyond existing feed-forward 3D approaches. We demonstrate leading detection accuracy, strong geometry reconstruction, and competitive rendering quality on the evaluated benchmarks, with substantial runtime gains.

1 Introduction

FIRE3D addresses the need to reconstruct complete, textured object-level scenes from cluttered, partially observed indoor captures. It combines feed-forward scene perception, compact object generation, and a large training corpus; from a 60-frame RGB video, it reconstructs up to twelve textured instances in under 60 seconds, including preprocessing.

1 Introduction

  • FIRE3D targets cluttered, partially observed scenes that require instance parsing, geometry completion beyond visible surfaces, appearance synthesis, and spatial-layout preservation.The paper motivates object-level digital twins for AR/VR, robotics, gaming, and content creation.Source: §1 · p. 1
  • Prior approaches leave a gap in fast feed-forward reconstruction of complete textured scenes from unsegmented, posed RGB-D observations estimated from raw RGB captures.The cited alternatives may omit complete geometry or texture, assume pre-segmented inputs, or depend on external systems or expensive optimization.Source: §1 · p. 2
  • FIRE3D lifts multi-view features into a 3D feature point cloud, predicts object validity, pose, and instance masks, then generates structure, shape, and material latents for each instance.Hierarchical VAEs decode complete textured meshes, which are assembled using predicted object poses.Source: §1 · p. 2, Figure 2 · p. 3
  • The training corpus comprises 80k scenes, 138,202 rendered sequences, 79,672 appearance-augmented sequences, and 500k object assets.It covers indoor layouts, object categories, occlusion patterns, and appearance variation for scene parsing, amodal completion, and textured reconstruction.Source: §1 · p. 2
  • From a 60-frame RGB video, FIRE3D reconstructs up to twelve textured instances in under 60 seconds, including Pi3X preprocessing.This timing is reported under the paper’s protocol.Source: §1 · p. 2

2 Related Works

Prior work offers scene-level reconstruction, feed-forward perception, or object reconstruction, but often lacks a combination of complete textured assets and integrated scene perception. FIRE3D combines feed-forward perception with batched textured reconstruction of detected objects.

2 Related Works

  • Scene-level methods can produce simulation-ready environments but rely on optimization, search, or iterative refinement; recent feed-forward models avoid test-time optimization but often need segmented inputs, prompts, external perception, or sequential object-wise inference.NeRF- and 3DGS-based approaches provide realistic novel-view synthesis but use fields or splats rather than editable object-level assets.Source: §2 · p. 3
  • FIRE3D distinguishes itself by integrating scene perception with batched feed-forward reconstruction of textured detected objects.The paper contrasts this with methods relying on optimization or iterative refinement and feed-forward systems that often need prompts, external perception, or sequential object-wise inference.Source: §2 · p. 3
  • Perception methods typically output boxes, masks, layouts, or partial geometry, while image-to-3D methods mainly target individual objects and DUSt3R successors do not explicitly produce editable object-level textured meshes and poses.FIRE3D instead reconstructs instances, poses, complete foreground geometry and texture, and a static background instance from unsegmented posed RGB-D observations.Source: §2 · p. 3

3 Method

FIRE3D reconstructs interactable textured scenes from posed RGB-D observations without per-scene optimization. Compact object latents and parallel generation underpin a unified perception-to-asset pipeline.

3.1 Representing an Object within 256 KBytes

  • HC-VAE reduces nominal grid feature capacity 32× while retaining high autoencoding fidelity; its two float32 grids occupy 256 KiB.At equal flow-training compute, HC-VAE also improves generation quality with faster sampling and decoding.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Instance-aware 3D Scene Perception from Posed RGB-D Observations

  • From posed RGB-D, FIRE3D voxelizes DINOv3-augmented 3D points, predicts object validity, instance masks and extents, then canonicalizes valid clouds for reconstruction.Source: §3.2 · pp. 4–5

3.3 Batched Point Cloud Conditioned Object Reconstruction and Scene Assembly

  • Cascaded flow matching generates shape latents before conditioned material latents for up to 16 objects; decoded meshes return to world coordinates, with structural surfaces grouped as background.Source: §3.3 · p. 5

3.4 Training

  • Training freezes DINOv3, trains perception on five annotated scene datasets, and packs up to 48 scene instances per flow-training batch.Source: §3.4 · p. 5

3.5 Inference

  • Inference filters proposals at validity 0.2 and OBB NMS IoU 0.4, then batches generation, decoding, and mesh conversion across instances.Source: §3.5 · p. 5

4 Experiments

FIRE3D is evaluated for video-based perception and reconstruction, single-image reconstruction, and downstream interactive-environment applications. Its experiments show leading detection and reconstruction results in several evaluated settings, alongside demonstrations in gaming and robotics.

4.1 Implementation Details

  • Training uses data from five scene datasets, appearance-augmented sequences, and 500k additional object assets; models are trained separately with augmentations and 12 flow-sampling steps.The training corpus includes 80k scenes and 138,202 rendered sequences, plus 79,672 appearance-augmented sequences.Source: §4.1 · p. 6

4.2 Experimental Settings

  • Evaluation covers posed-RGB-D video perception, video reconstruction with ground-truth or inferred perception, and single-image reconstruction from a Pi3X-estimated point map.Datasets and scene/object assets are excluded from training; real-world tests cover perception and object geometry, while rendering comparisons use synthetic data.Source: §4.2 · pp. 6–7

4.3 Experimental Results

  • FIRE3D leads mAP on all three perception datasets and mIoU on iTHOR and Imaginarium, with the fastest reported inference.The comparison reports detection and segmentation metrics and average inference time across datasets.Source: §4.3 · p. 8, Table 2 · p. 8
  • With ground-truth perception, FIRE3D leads all geometry metrics on iTHOR; on Imaginarium it leads CD, while ShapeR performs best on the real-world benchmark.Under inferred perception, FIRE3D improves all reported geometry metrics over the ShapeR pipeline.Source: §4.3 · p. 8, Table 3 · p. 8
  • On the held-out 3D-Front split, FIRE3D achieves the best CD, F1, and NC among the compared single-image methods.FIRE3D takes a Pi3X-estimated point map from a single RGB image as input.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 Interactive Environment Applications

  • Independent assets support rapid game editing and integration, Blender water effects, and IsaacSim/IsaacLab pick-and-place demonstrations; physical stability and sim-to-real transfer remain unevaluated.The demonstrations include a virtual shooting game and a robot placing an object from a chair onto a table.Source: §4.4 · pp. 8–9

5 Conclusion & Limitation

FIRE3D reconstructs object-level textured scenes from posed RGB-D observations estimated from RGB captures, using feed-forward perception and object-centric generation. Its scope is static indoor scenes, and its output quality depends on input depth, camera poses, scale, and instance parsing.

5 Conclusion & Limitation

  • FIRE3D combines instance-aware perception and object-centric generation with a compressed object representation to batch-reconstruct independently editable assets with amodal geometry and texture.The network interface uses unsegmented, posed RGB-D observations estimated from single-image or casual-video RGB captures.Source: §5 · p. 9
  • The framework focuses on static indoor scenes and requires posed RGB-D at its network interface, with external geometric preprocessing for RGB-only captures.Reconstruction quality depends on depth, camera poses, scale, and instance parsing.Source: §5 · p. 9
  • Watertightness, physical stability, relightability, and articulation are not guaranteed and are left to future work.Source: §5 · p. 9

A More Experiment Results

FIRE3D is compared with reconstruction and perception systems across video and single-image settings, with additional tests of latent compression, preprocessing sensitivity, and runtime. The results show substantial speed gains, while quality advantages depend on the baseline and metric.

Single-image Reconstruction

  • On 3D-Front single-image tests, FIRE3D achieves better geometry performance than Gen3DSR, MIDI, and SceneGen, with coherent geometry and scene layouts in qualitative examples.Evaluation uses 20 test images and aligns each predicted scene to its reference with one similarity transform.Source: A More Experiment Results · p. 15, Figure A4 · p. 15

Ablation Study

  • With Pi3X poses and depth, mAP falls from 0.54 to 0.46 and CD rises from 7.98 to 8.99 cm, while PSNR remains similar.The ablation uses 20 randomly selected iTHOR and Imaginarium scenes; batched and sequential execution match in the tested configuration, with batching over 10× faster.Source: A More Experiment Results · p. 15, Table A1 · p. 15

Comparison against HoloScene

  • Against HoloScene, FIRE3D improves scene-level CD and F1 and object-level CD and F1, while HoloScene scores higher on scene-level NC and PSNR.HoloScene takes approximately eight hours per scene for optimization alone; FIRE3D takes approximately one minute, a 480× speedup.Source: A More Experiment Results · p. 16, Table A2 · p. 16

Comparison against SimRecon and LiteReality

  • Against SimRecon, FIRE3D improves mAP on both datasets and overall mIoU, but trails on Imaginarium mIoU, 0.52 versus 0.56.Against LiteReality under matched GT detection and segmentation, FIRE3D improves all three geometry metrics and runs 7.46× faster for geometry inference and mesh post-processing.Source: A More Experiment Results · pp. 16–17, Table A3 · p. 16, Table A4 · p. 17

Comparison against composed pipeline

  • On 30 Imaginarium scenes, FIRE3D improves all reported quality metrics over the composed pipeline and achieves 6.52× faster per-object network inference.Mesh post-processing is also faster, but the system-level gains cannot be attributed to HC-VAE alone.Source: A More Experiment Results · p. 17, Table A5 · p. 17

VAE Reconstruction Comparisons

  • HC-VAE compresses the object latent by 32× while retaining high geometry and rendering reconstruction quality, with a small change in autoencoding fidelity.The comparison measures encoding–decoding distortion, not the effect of latent resolution on flow generation.Source: A More Experiment Results · p. 17, Table A6 · p. 18

Flow generation under equal training compute

  • Under matched flow-training compute, HC-VAE improves every reported geometry and rendering metric; sampling and decoding take 0.392 s per object versus approximately 4 s for uncompressed models.These times exclude perception and asset export; the uncompressed models had not converged at the matched budget.Source: A More Experiment Results · p. 18, Table A7 · p. 18

Runtime Analysis and Failure mode analysis

  • Textured inference, decoding, and export take 4.783 s per object; with Pi3X preprocessing, twelve instances total 58.4 s, while geometry-only processing supports thirty instances per minute.FIRE3D can miss objects when perception fails to detect them, and point-based conditioning limits perfect shape and texture reconstruction.Source: A More Experiment Results · p. 19, Table A8 · p. 19

B.1 Data Preparation Details

The training data combines rendered indoor scenes with appearance-augmented videos and additional object assets. Rendering supplies posed RGB-D observations under varied camera settings.

Data curation

  • The corpus draws indoor scenes from SAGE-10k, InternScenes, ProcTHOR, MansionWorld, and SceneSmith, totaling 80k scenes and 138,202 rendered videos.The rendered videos use randomized camera intrinsics.Source: §B.1 · p. 19
  • Flux.2 adds 79,672 appearance-augmented videos, yielding 217,874 sequences that share the underlying scenes.Source: §B.1 · p. 19

Data curation and Data rendering

  • An additional 500k objects from four object datasets are used in training the flow-matching reconstruction model.Blender rendering produces 512×512 RGB-D frames with metric depth, intrinsics, and camera-to-world poses; camera FOV ranges from 40° to 90°.Source: §B.1 · p. 19

B.2 Data Augmentation Details

Training augmentations target scene orientation, imperfect geometric inputs, and the gap between synthetic renders and realistic captures. They include rotations, injected pose/depth noise, and Flux.2-generated appearance variation.

Scene rotation

  • Random 90°, 180°, and 270° rotations about the z-axis expose the perception model to varied object orientations.Source: §B.2 · p. 19

Noise-based augmentation

  • Gaussian noise simulates imperfect preprocessing estimates, using standard deviations of 0.02 m for translation and depth and 0.01° for camera rotation.The motivation is that synthetic poses and depth are more precise than practical estimates.Source: §B.2 · pp. 19–20

Flux-based augmentation

  • Flux.2 generates 79,672 videos of photorealistic images from synthetic renders to mitigate the synthetic-to-real appearance gap.Source: §B.2 · p. 20, Figure A9 · p. 20

B.3.1 Hierarchical VAEs design

HC-VAE further compresses SC-VAE latents to support efficient multi-object generation. The accompanying batchified decoding and export stages process objects together while preserving separate per-object outputs.

B.3.1 Hierarchical VAEs design

  • HC-VAE encodes SC-VAE sparse features with a lightweight sparse 3D convolutional encoder and decoder to produce a more compact latent representation.Source: §B.3.1 · p. 20
  • The shape HC-VAE compresses 32-channel sparse shape features to a 64-channel latent and predicts subdivision signals for sparse-structure refinement.Its sparse 3D U-Net-style decoder reconstructs the 32-channel shape feature field.Source: §B.3.1 · p. 20
  • The material HC-VAE maps 32-channel sparse material features to a 64-channel latent and reconstructs material features without predicting subdivision.Material attributes are decoded on the given sparse support.Source: §B.3.1 · p. 20

B.3.2 Hierarchical Batchified Object Decoding

  • Batchified flow inference packs object points, features, instance indices, and canonical transforms so one forward pass denoises multiple object latents while retaining object-specific conditioning.Sparse coordinates and corresponding latent features are packed jointly with disjoint object supports for feature and material generation.Source: §B.3.2 · p. 21
  • Mesh post-processing pads variable-size meshes with masks for joint GPU processing, while batched UV rasterization and material sampling produce separate texture maps for each object.The meshes are recovered, transformed back to world coordinates, and exported independently.Source: §B.3.2 · p. 21

B.4 Architecture Details

FIRE3D combines point-based scene perception and instance decoding with point-cloud-conditioned flow models for object reconstruction. The perception module predicts instance properties and segmentation, while three flows generate sparse structure, shape, and material features.

B.4.1 Instance-aware 3D scene perception model

  • Dense DINOv3 descriptors are lifted into 1024-dimensional point features, processed by a point-based U-Net, and scene context is capped at 30k tokens.Sparse input and segmentation features are not capped at 30k.Source: §B.4.1 · p. 21
  • A transformer decoder with 384 candidate slots predicts object validity, translation, rotation, scale, and point-to-instance segmentation.The architecture uses six layers each for global reasoning, segmentation refinement, and object-level decoding.Source: §B.4.1 · p. 21

B.4.2 Point cloud conditioned generative reconstruction model

  • Three point-cloud-conditioned flow models reconstruct sparse structure, shape features, and material features using a shared conditioning design and 1536-width, 30-block backbones.Object-level point features are encoded with Fourier positional features and projected to a 1024-dimensional context.Source: §B.4.2 · pp. 21–22
  • The structure flow predicts 8-channel latents on a 23 grid, while shape and material flows predict 64-channel latents on sparse support.Shape and material flows operate at sparse resolution 8; decoding uses resolution 512.Source: §B.4.2 · p. 22

B.5 Training Details

The perception and generative reconstruction models are trained separately from scratch with different schedules and precision settings. Their training setups also differ in gradient clipping, EMA use, and gradient accumulation.

B.5 Training Details

  • Perception training runs for 400k steps with AdamW at learning rate 10−5, float32 precision, gradient clipping at norm 0.1, and no EMA.The final 400k-step checkpoint is evaluated.Source: §B.5 · p. 22
  • Flow-model training runs for 200k steps with AdamW at learning rate 10−4, bfloat16 mixed precision, gradient clipping at norm 1.0, and EMA decay 0.9999.Gradient accumulation is 2/1/2 steps for structure/shape/material, and inference uses non-EMA weights.Source: §B.5 · p. 22

Related papers

Loading 2609.08848v2…