PaperlayerView exact revision on arXiv

Source-linked AI summary

Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views

Mijin Yoo, In Cho, Subin Jeon, Jiwoo Lee, Eunbyung Park, Seon Joo Kim

arXiv:2606.29513v1cs.CVcs.GR

TL;DR

Feed-forward 3D reconstruction typically represents scenes as unstructured primitives, leaving object-level structure to be recovered afterward. This paper predicts instance-structured token groups from unposed multi-view images, surpassing per-scene optimization baselines in instance segmentation while remaining competitive in reconstruction quality.

  • Problem

    Feed-forward reconstructions represent scenes as dense, unstructured primitives, leaving object identities and boundaries scattered across elements rather than as first-class units.

  • Method

    The model decomposes unposed multi-view images into groups pairing instance tokens for object identity with anchor tokens for local geometry and appearance, decoded into 3D Gaussians.

  • Results

    The feed-forward model surpasses per-scene optimized baselines in class-agnostic instance segmentation while achieving competitive reconstruction quality.

  • Takeaways & Limitations

    The token groups provide a direct object-level interface for 3D scene understanding, editing, and open-vocabulary instance retrieval without post-hoc processing.

  • Takeaways & Limitations

    The evaluation focuses primarily on bounded indoor scenes, while outdoor and large-scale environments, complex instances, and dynamic scenes remain open challenges.

Abstract

from arXiv · show

A 3D scene is understood through its objects, not the primitives that compose them. Yet feed-forward reconstruction methods output dense, unstructured sets of points or Gaussians, leaving object-level structure to be recovered after the fact. We propose a feed-forward framework that decomposes a scene into instance-structured 3D token groups directly from unposed multi-view images -- compact object-centric units from which reconstruction, segmentation, and manipulation all follow. Each token group pairs an instance token capturing entity-level identity with anchor tokens that encode local geometry and appearance, which are decoded into a set of 3D Gaussians. This two-level factorization decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product. The token groups are learned through differentiable rendering with joint reconstruction and segmentation supervision, requiring no 3D annotations. Our feed-forward model surpasses per-scene optimization baselines in class-agnostic instance segmentation while remaining competitive in novel view synthesis. Beyond these metrics, the same token groups directly unlock instance-level scene editing -- removing, translating, or inserting objects by operating on their groups -- as well as efficient open-vocabulary 3D instance retrieval, where retrieval complexity scales with the number of instances rather than primitives.

1 Introduction

The paper argues that feed-forward 3D reconstruction should represent object instances as first-class units rather than unstructured primitives. It introduces instance-structured token groups learned from unposed multi-view images, enabling reconstruction, segmentation, semantic retrieval, and object-level editing.

  • Motivation: Feed-forward reconstruction methods produce dense points or Gaussians without representing which primitives belong to the same object.Attaching 2D foundation-model features to primitives adds local annotation but leaves object information scattered across primitives.
  • Representation: The proposed representation decomposes each scene into compact instance-structured 3D token groups in one forward pass.Each group combines an instance token summarizing object identity and extent with anchor tokens encoding local geometry and appearance, which decode into 3D Gaussians.
  • Semantics: Token groups store shared instance-level embeddings with lightweight anchor-level residuals for compact, text-retrievable semantic representations.This distills 2D foundation-model features into groups while preserving spatially varying detail.
  • Results and capabilities: The model surpasses per-scene optimized baselines in class-agnostic instance segmentation while achieving competitive reconstruction quality.The same token groups also support removing, translating, or inserting objects and open-vocabulary 3D instance retrieval.

2 Related work

Prior feed-forward 3D Gaussian methods predict renderable Gaussians from input views, but their representation units remain low-level primitives. Existing instance-aware and object-centric approaches add identity or entity structure, yet often rely on scene-specific optimization, motivating token groups that natively organize local Gaussians into instances.

  • Feed-forward 3D Gaussian reconstruction: Feed-forward methods predict 3D Gaussians directly from input views, removing the original per-scene optimization requirement of 3D Gaussian Splatting.pixelSplat and MVSplat assume calibrated views, while geometry foundation models enable pose-free reconstruction.
  • Feed-forward 3D Gaussian reconstruction: These methods use low-level units such as pixels, point-map elements, Gaussian queries, or individual Gaussians, which are suited to rendering but not human-aligned scene understanding.The proposed representation instead uses anchor tokens to generate local Gaussians and instance tokens to group anchors into instances.
  • Semantics and instances for multi-view 3D scenes: Instance-aware 3DGS methods attach identity information to Gaussians or object-ID channels, but require scene-specific reconstruction and labeling.Gaussian Grouping optimizes per-Gaussian identity features, while ObjectGS uses onehot object-ID channels inherited from object-aware anchors.
  • Object-centric 3D scene reconstruction: Object-centric reconstruction models scenes as coherent entities, including slot-based and set-prediction models, SlotLifter, and GOCL.SlotLifter learns object-centric radiance fields, while GOCL provides object-centric supervision for per-scene optimized 3DGS using a scene-agnostic object codebook.

3 Method

The method tokenizes unposed multi-view images into instance-structured groups, combining group tokens for object identity with anchor tokens that decode local geometry and appearance into 3D Gaussians. It learns reconstruction, grouping, and decomposed semantic features through differentiable rendering with only 2D supervision.

  • Multi-view feature encoding: A frozen 3D foundation model fuses multi-view features, pointmaps, RGB patches, and pointmap patches into context tokens for token-group decoding.Patch-aligned pointmaps provide 3D coordinates, while flattened fused features represent each view-and-patch context element.
  • Token group decoding: Two cross-attention transformers decode image-grounded anchor tokens and object-level group tokens, whose softmax similarities assign anchors competitively to instances.Anchor tokens reconstruct Gaussian primitives, while group tokens act as grouping queries; a void channel accounts for non-instance regions.
  • Gaussian reconstruction: Each decoded anchor generates Ng 3D Gaussians with predicted offsets, scale, rotation, opacity, and spherical harmonics, inheriting its parent anchor’s group assignment.The resulting L instance-level groups can be rendered and manipulated independently.
  • Training objectives: Training uses 2D reconstruction and instance-mask supervision: rendered Gaussians match target RGB images, while Hungarian-matched probability maps receive Dice and BCE losses.The total objective is L = Lrender + λseg Lseg, with linear warm-up for λseg during early training.
  • Semantic token encoding: Semantic representation decomposes each instance into a shared group embedding sℓ∈RD and a low-dimensional anchor residual rk∈Rd, with d ≪D.Feature reconstruction and group-level alignment make group embeddings capture object-level semantics while residuals model within-instance variation.

4 Experiments

Experiments on ScanNet evaluate the token groups for reconstruction, feature lifting, class-agnostic instance segmentation, manipulation, and open-vocabulary 3D instance retrieval. The framework achieves strong segmentation and feature-lifting results while enabling direct instance-level operations and efficient retrieval.

  • Evaluation Setup: The ScanNet evaluation covers feed-forward novel-view reconstruction, open-vocabulary feature lifting, and class-agnostic instance segmentation, with qualitative tests of manipulation and 3D instance retrieval.Reconstruction uses PSNR, SSIM, and LPIPS; feature lifting uses mIoU and pixel accuracy; instance segmentation uses AP, AP50, and AP25.
  • Reconstruction and Feature Lifting: The model achieves the best feature-lifting mIoU on source and target views, while reducing semantic storage from 8.4M scalars in Uni3R to 59.4K.The representation concentrates semantic information at the instance level rather than storing features independently at every Gaussian.
  • Class-Agnostic Instance Segmentation: The feed-forward model achieves the best class-agnostic instance segmentation across all AP metrics, surpassing Gaussian Grouping, ObjectGS, and IGGT + LUDVIG.The comparison includes per-scene optimization baselines and a feed-forward method with post-hoc optimization; training uses only 2D supervision.
  • Token Group Manipulation: Token groups enable group-wise rendering, removal, insertion, and rigid transformation, with edits localized to the selected instance without masks, post-hoc processing, or per-scene optimization.The operations act directly on the selected group’s tokens and associated Gaussians, leaving neighboring objects and the background unaffected.
  • Open-Vocabulary 3D Instance Retrieval: Open-vocabulary retrieval matches text or image queries against shared group-level embeddings, scaling linearly with fewer than 100 instances rather than 131,072 Gaussians in pixel-aligned baselines.Each group stores a semantic embedding lifted from a 2D foundation model, enabling instance-level retrieval without post-processing.
  • Ablation Studies: Joint training is necessary: the sequential variant reduces segmentation AP from 0.193 to 0.032 and substantially degrades reconstruction and segmentation performance.The ablations compare sequential training with joint training and a joint variant without λseg warm-up.

5 Discussion

The discussion positions instance-structured token groups as a native interface for future compositional 3D reasoning and object-centric robotics, while identifying scalability and expressivity limitations. Current evaluation centers on bounded indoor scenes, with larger environments and more complex instance semantics remaining open challenges.

  • Discussion: The representation treats object instances as a native interface instead of recovering structure after reconstructing dense primitives.This reframes 3D scene representation around object instances and motivates the extensions discussed in this section.
  • Toward compositional reasoning and generation: Compact object-aligned token groups could connect 3D scenes to large models by supporting entity-level reasoning with finer detail from anchor tokens.The proposed direction uses group-level tokens for entities and anchor tokens when more detailed scene information is needed.
  • Toward object-centric world models for robotics: For robotics, token groups could provide compact instance-level handles from a handful of unposed images that map onto objects an agent must reason about and manipulate.The passage frames these handles as a bridge between scene perception and object-centric world models.
  • Limitations: Scaling beyond bounded indoor scenes remains open because outdoor and large-scale environments may require revisiting the fixed upper bound of L = 100 groups and model training.The limitation concerns both the representation’s group capacity and the training regime at larger scale.
  • Limitations: A single shared group-level token may insufficiently express complex or highly varied instance semantics.The passage suggests multiple shared semantic tokens as basis features within each group as a possible remedy.

6 Conclusion · A Societal impacts

The framework represents 3D scenes as instance-structured token groups, with identity and local geometry/appearance supporting reconstruction, understanding, manipulation, and retrieval. Object-level editing benefits applications but raises privacy and fabrication risks, motivating provenance, consent, edit disclosure, and manipulation detection.

  • 6 Conclusion: The framework reconstructs 3D scenes as instance-structured token groups, making object instances first-class in the representation.This removes the need to recover object structure after reconstruction.
  • 6 Conclusion: Each token group combines an instance token for entity-level identity with anchor tokens for local geometry and appearance.The two token levels separate entity identity from local scene properties.
  • 6 Conclusion: The token groups support accurate reconstruction and scene understanding while directly enabling instance-level manipulation and retrieval without post-hoc processing.The representation serves as a native interface for these capabilities.
  • A Societal impacts: Instance-structured 3D representations can benefit robotics, AR/VR, and content creation by enabling object-level scene editing.Their object-level structure makes reconstructed scenes editable at the level of individual objects.
  • A Societal impacts: Reconstructing and recomposing real environments from casual captures raises privacy concerns.The concern arises because casual captures can encode real environments in editable 3D representations.
  • A Societal impacts: Token-level editing lowers the barrier to fabricated 3D scenes, creating risks alongside the framework’s capabilities.The paper identifies fabrication as a societal risk of easier 3D editing.
  • A Societal impacts: The authors encourage provenance signals, consented capture, and disclosure of edits, while identifying manipulated-3D-content detection as an important complement.These measures address provenance, consent, transparency, and detection of edited content.

A.1 Additional implementation details

The implementation uses 1024-dimensional features, processes and renders 256 × 256 images, and predicts Gaussian attributes relative to anchor centers with degree-2 spherical harmonics.

  • Feature dimensions are set to 1024, with 256 × 256 images used for both input and rendering.
  • The Gaussian head predicts scale, rotation, opacity, local offsets, and degree-2 spherical harmonics coefficients for each anchor.Local offsets are defined relative to the anchor center.
  • For 256 × 256 inputs, the VGGT backbone changes the initial DINO patch size from 14 to 16 and follows Uni3R.

B Experiment setup details · B.1 ScanNet class-agnostic novel-view instance segmentation

The ScanNet evaluation uses a shared, interleaved multi-view setup with COLMAP initialization and test-view rendering. It compares per-scene Gaussian labeling baselines with an IGGT + LUDVIG implementation using mask-regularized 3DGS.

  • B.1 ScanNet class-agnostic novel-view instance segmentation: The evaluation follows the 40-scene ScanNet test subset introduced by LSM.Each scene is sampled from its full sequence for reconstruction and evaluation.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: Frames are sampled from each full sequence with stride 10.The sampled frames provide the views used to reconstruct scene-level cameras and form the evaluation split.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: Scene-level COLMAP cameras are reconstructed from the sampled frames.All compared methods use this shared camera initialization.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: An interleaved split assigns 8 training views and 7 test views among the sampled frames.Evaluation is conducted on the test views.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: Gaussian Grouping and ObjectGS are trained per scene using the shared COLMAP initialization.Their predictions are rendered on the same test views for evaluation.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: For Gaussian Grouping and ObjectGS, AP confidence is computed from rendered per-pixel scores.This defines how confidence values are obtained for both per-scene Gaussian labeling baselines.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: IGGT + LUDVIG uses mask-regularized 3DGS instead of vanilla 3DGS for label uplifting.The backbone follows PanSt3R’s panoptic regularization strategy to reduce Gaussian spreading across object boundaries.
  • B.1 ScanNet class-agnostic novel-view instance segmentation: IGGT instance masks are used as input labels and converted into one-hot instance labels before uplifting.The supplied passage states that these labels are then uplifted to the mask-regularized 3DGS representation.

C Additional generalization experiments

This section evaluates the tokenization framework beyond its in-domain setting through training on RealEstate10K with SAM2 pseudo-labels and zero-shot transfer from ScanNet to MipNeRF360, using two-view inputs in both studies.

  • Generalization studies: The generalization experiments comprise RealEstate10K training with SAM2 pseudo-labels and zero-shot transfer of a ScanNet-trained model to MipNeRF360.The RealEstate10K study replaces human-annotated masks with SAM2 pseudo-labels.
  • Experimental setting: Both generalization experiments use the 2-view input setting.

C.1 RealEstate10K with SAM2 pseudo-labels … D.2 Additional feature lifting results

Across additional experiments and qualitative analyses, the instance-structured tokenizer remains effective with automatically generated supervision, transfers zero-shot across datasets, and produces coherent reconstruction and feature-lifting results.

  • C.1 RealEstate10K with SAM2 pseudo-labels: RealEstate10K evaluation uses two sampled source views per clip and held-out target views, with no instance annotations in the dataset.The dataset contains casually captured real-estate videos with camera poses but no instance masks.
  • C.1 RealEstate10K with SAM2 pseudo-labels: SAM2-generated masks supervise 2D instance segmentation without human-annotated 2D or 3D labels, although identity tracks can break after occlusion or re-entry.Broken tracks may assign a new identity when an object briefly leaves or re-enters view.
  • C.1 RealEstate10K with SAM2 pseudo-labels: Despite noisy SAM2 supervision, the tokenizer outperforms C3G across all three RealEstate10K reconstruction metrics, supporting anchor–group tokenization under automatically generated masks.SAM2 masks shown on test views are reference supervisory signals, not evaluation targets.
  • C.2 Zero-shot transfer to MipNeRF360: Our model outperforms Uni3R on PSNR, SSIM, and LPIPS under zero-shot transfer, indicating that anchor–group tokenization captures a representation prior beyond the training distribution.The result holds despite the distribution shift from ScanNet to MipNeRF360.
  • D Additional qualitative results: Additional qualitative comparisons cover three more ScanNet scenes for novel-view reconstruction, open-vocabulary feature lifting, and class-agnostic instance segmentation.These examples complement the main-paper figures.
  • D.1 Additional reconstruction results: Additional reconstruction views reproduce overall scene structure and major object appearances faithfully, but sometimes smooth fine high-frequency details relative to pixel-aligned baselines.This qualitative behavior is consistent with the small reconstruction gap reported in Table 1.
  • D.2 Additional feature lifting results: Additional feature-lifting results yield semantic maps that respect object boundaries and remain consistent across regions belonging to the same entity.The decomposed group-level and anchor-level features support the source-view and target-view mIoU gains reported in Table 1.

D.3 Additional class-agnostic instance segmentation results

Additional ScanNet comparisons show that token-group-based segmentation maintains coherent, class-agnostic instance boundaries across more scenes, avoiding fragmentation seen in per-Gaussian identity baselines, especially on large contiguous surfaces.

  • Additional qualitative results: Token-group-based segmentation produces coherent instance boundaries across three additional ScanNet scenes, unlike fragmented per-Gaussian identity baselines.The comparison extends the qualitative evaluation from Figure 5 of the main paper.
  • Additional qualitative results: The improvement is particularly evident on large, contiguous surfaces such as walls, floors, and beds.These surfaces are identified as especially prone to fragmentation in per-Gaussian identity baselines.
Loading 2606.29513v1…