PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural USD: An object-centric framework for iterative editing and control

Alejandro Escontrela, Shrinu Kushagra, Sjoerd van Steenkiste, Yulia Rubanova, Aleksander Holynski, Kelsey Allen, Kevin Murphy, Thomas Kipf

arXiv:2510.23956v1cs.CVcs.AI

TL;DR

Existing conditioning approaches can be model-specific and restrict users to global scene edits, limiting portable, incremental object editing. Neural USD represents per-object appearance, geometry, and pose and fine-tunes models using paired video frames to disentangle control signals. It supports iterative edits that preserve other scene elements and outperforms Object 3DIT and Chained, improving over Neural Assets.

  • Problem

    Conditioning approaches can be model-specific or limit users to global scene edits, restricting portable object-level incremental updates.

    Source: §1 · p. 2

  • Method

    Neural USD encodes per-object appearance, geometry, and pose, and fine-tunes models on paired video frames to disentangle control signals.

    Source: §1 · pp. 2–3

  • Results

    Neural USD supports iterative edits that keep other scene elements unchanged or consistent, outperforms Object 3DIT and Chained, and improves over Neural Assets.

    Source: §4.2 · p. 7, §4.3 · p. 8

  • Takeaways & Limitations

    The paper presents Neural USD as a flexible, portable conditioning standard for generative modeling.

    Source: §5 · p. 9

    Obtaining 3D bounding-box annotations at scale remains an open challenge.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Amazing progress has been made in controllable generative modeling, especially over the last few years. However, some challenges remain. One of them is precise and iterative object editing. In many of the current methods, trying to edit the generated image (for example, changing the color of a particular object in the scene or changing the background while keeping other elements unchanged) by changing the conditioning signals often leads to unintended global changes in the scene. In this work, we take the first steps to address the above challenges. Taking inspiration from the Universal Scene Descriptor (USD) standard developed in the computer graphics community, we introduce the "Neural Universal Scene Descriptor" or Neural USD. In this framework, we represent scenes and objects in a structured, hierarchical manner. This accommodates diverse signals, minimizes model-specific constraints, and enables per-object control over appearance, geometry, and pose. We further apply a fine-tuning approach which ensures that the above control signals are disentangled from one another. We evaluate several design considerations for our framework, demonstrating how Neural USD enables iterative and incremental workflows. More information at: https://escontrela.me/neural_usd .

1 INTRODUCTION

Existing conditioning approaches can be architecture-specific and limit users to global edits, making portable, incremental object editing difficult. Neural USD structures per-object control signals and uses fine-tuning to support iterative edits while preserving other scene elements.

1 INTRODUCTION

  • Conditioning approaches may be model-specific or cause global scene changes, limiting portable, object-level incremental editing.Local conditioning edits can compound global changes across multiple edits, while text prompts can struggle to express complex layouts.Source: §1 · p. 2
  • Neural USD is an object-centric format for controlling object position, appearance, and geometry across a broad class of generative models.The framework represents images as assets with appearance, geometry, and pose.Source: §1 · p. 2, Figure 2 · p. 2
  • Fine-tuning existing models on paired video frames learns disentangled control signals across modalities.The framework aims to keep appearance and geometry pose-invariant.Source: Figure 2 · p. 2, §1 · p. 3
  • Neural USD supports serial edits that change target objects while keeping other scene elements fairly unchanged or consistent.The introduction reports comparisons against standard baselines and more precise object control measured by reconstruction error.Source: §1 · p. 3

2 RELATED WORK

Prior work explores object-centric representations, personalization, spatial control, and 3D-aware generation, but the described approaches have limitations in real-world data, spatial control, or editing. Neural USD combines 2D and 3D spatial conditioning with appearance and geometry inputs.

2 RELATED WORK

  • Object-centric learning methods represent scenes as distinct entities and support attribute manipulation, motion modeling, and 3D pose estimation, but struggle with real-world data.The paper positions self-supervised visual encoders and pre-trained diffusion models as its route to object-centric learning in realistic settings.Source: §2 · p. 3
  • Personalized generation methods often produce single-subject images without spatial control; multientity methods offer limited control, and Subject-Diffusion lacks 3D pose control.Source: §2 · p. 3
  • 3D-aware generation spans explicit 3D representations and diffusion-based methods, with some recent work targeting multi-object real-world scenes.Source: §2 · p. 3
  • OBJect-3DIT is described as synthetic-data limited, while LooseControl uses 3D bounding boxes as depth maps for pose control but is not directly applicable to editing.Source: §2 · pp. 3–4
  • Spatial-control methods use prompts, attention, latent editing, or fine-tuned conditioning, while Neural USD unifies 2D and 3D spatial conditioning with appearance and geometry inputs.The paper describes several existing methods as controlling one scene aspect at a time.Source: §2 · pp. 3–4

3 METHOD

Neural USD represents scenes as object-centric assets with separate appearance, geometry, and pose signals, then combines those signals to condition image models. The method uses video image pairs and modality dropout to learn more independent controls for iterative object editing.

3.1 DATA

  • Training data uses readily available 2D and 3D annotations, while obtaining 3D bounding-box annotations at scale remains an open challenge.The dataset is composed by applying separate annotation models, including depth estimation and box-conditioned object masking.Source: §3.1 · p. 4

3.2 ASSETS IN NEURAL USD

  • Each Neural USD asset is a tuple of object attributes, including pose, appearance, geometry, and optionally text descriptors.The scene representation is a list of assets, allowing multiple objects and modalities to be represented together.Source: §3.2 · pp. 4–5

3.3.1 APPEARANCE AND GEOMETRY ENCODING

  • Appearance and geometry are encoded as continuous tokens with separate modality encoders, and per-object feature processing removes global correlations between objects.The paper reports that per-object depth normalization improves generalization, while a shared image-depth backbone gave suboptimal results.Source: §3.3.1 · p. 5, §3.3 · p. 5

3.3.2 2D AND 3D POSE ENCODING

  • Separate 2D and 3D pose encodings provide simple in-image dragging and richer control over camera distance and rotation, respectively.The 2D pose uses image-normalized box coordinates; the 3D representation projects bounding-box corners to the image plane and includes depth.Source: §3.3.2 · p. 5

3.4 COMBINING ENCODINGS

  • Appearance, geometry, and pose tokens are concatenated into asset encodings, which are then concatenated across assets for downstream conditioning.The background can be represented as another asset; masking foreground objects in background signals improved results.Source: §3.4 · pp. 5–6

3.5 FINE-TUNING MODELS WITH NEURAL USD

  • The encoders and Stable Diffusion v2.1 are fine-tuned end to end, with modality dropout intended to make modality features invariant to other modalities.The authors use Stable Diffusion v2.1 as an example model and do not expect state-of-the-art prediction quality from it.Source: §3.5 · p. 6

3.6 LEARNING FROM PAIRS OF IMAGES

  • Training from video pairs combines source-image appearance and geometry with target-image pose to reconstruct the target, addressing pose entanglement from single annotated images.The training objective uses Stable Diffusion v2.1's denoising diffusion loss; the resulting model can be controlled through multiple independent signals.Source: §3.6 · p. 6

4 EXPERIMENTS

The experiments assess object-level pose, appearance, and geometry control, iterative edits that preserve other scene elements, and comparisons with other conditioning approaches. Across the reported comparisons, Neural USD performs better than the evaluated baselines.

4 EXPERIMENTS

  • Experiments use MOVi-E, Objectron, Waymo Open, and EgoTracks, with objects represented by 2D and 3D bounding-box annotations.Objects with small boxes are filtered out, and images are randomly flipped.Source: §4 · p. 7

4.1 POSE, APPEARANCE AND GEOMETRIC CONTROL

  • Neural USD supports translating, rotating, and scaling objects, geometry-only conditioning, object replacement, and background replacement while other scene elements remain unchanged.The experiments also demonstrate conditioning on geometry to generate novel appearances that satisfy the original object's 3D structure.Source: §4.1 · p. 7

4.2 ITERATIVE EDITING

  • Across iterative edits, other scene elements stay unchanged or change consistently with the original; the examples combine pose, appearance, geometry, and background edits.Figure 6 shows changing a glass's appearance, replacing it with a bottle, and then editing pose, appearance, and background together.Source: §4.2 · pp. 7–8, Figure 6 · p. 8

4.3 COMPARISON AGAINST OTHER MODELS

  • On MOVi-E, Objectron, and Waymo, Neural USD outperforms Object 3DIT and Chained and improves over Neural Assets.The comparison evaluates reconstruction of target images in scenes with multi-object changes.Source: Figure 7 · p. 8, §4.3 · p. 8
  • The evaluation measures target-image reconstruction with SSIM, LPIPS, and PSNR, and compares reconstruction and controllability across methods.For controllability, an input describes the difference between source and target scenes, and the predicted image is compared with the target.Source: §4.3 · pp. 8–9
  • Neural USD outperforms the compared models on reconstruction and controllability.Controllability is difficult to assess for methods without an interface for editing from input conditioning signals.Source: §4.3 · p. 9

5 CONCLUSION

Neural USD is an object-centric conditioning framework for controlling appearance, geometry, and pose. The authors report superior performance in structured scene synthesis and object control, and identify scaling the approach as future work.

5 CONCLUSION

  • The authors report superior performance in structured scene synthesis and object control.Figure 8 compares reconstruction and controllability on MOVi-E, Objectron, and Waymo; the supplied caption gives no numerical results.Source: §5 · p. 9, Figure 8 · p. 9
  • Neural USD provides object-centric control over appearance, geometry, and pose.The conclusion describes iterative edits that leave other scene elements consistent and unchanged.Source: §5 · p. 9
  • Future work includes using a larger, more modern base model and scaling to ImageNet-size datasets and beyond.The current approach uses Stable Diffusion.Source: §5 · p. 9

A DATASETS

The appendix describes four datasets and dataset-specific annotation or filtering choices used in the experiments.

A.1 EGOTRACKS

  • EgoTracks samples exclude small boxes, use source and target frames within 15 frames, and filter motion blur or extreme background shift.The small-box threshold is one tenth of the normalized screen height or width.Source: §A.1 · p. 17

A.2 OBJECTRON

  • Objectron’s 15,000 clips cover nine object categories; 2D boxes are derived by projecting 3D-box corners and enclosing them tightly.Source: §A.2 · p. 17

A.3 WAYMO OPEN

  • Waymo uses front-view clips and car boxes; because provided 2D and 3D boxes are misaligned, 2D boxes are reprojected from 3D boxes.Only yaw rotation is available, so the other rotation angles are set to zero; the misalignment makes paired-frame training infeasible.Source: §A.3 · p. 17

A.4 MOVI-E

  • MOVi-E contains 10,000 simulated videos with 11 to 23 objects per scene, collisions, randomly sampled environment lighting, and linear camera motion.Source: §A.4 · p. 17

B.1 OBJECT 3DIT

The appendix describes the comparison methods and how they are evaluated or adapted for the experiments. Their stated capabilities and limitations differ across editing, conditioning, and reconstruction tasks.

B.1 OBJECT 3DIT

  • Object 3DIT is fine-tuned from Zero-1-to-3 for scene-level 3D editing, with instructions derived from target-object translation and rotation.It lacks support for significant viewpoint changes because it does not encode camera poses.Source: §B.1 · pp. 17–18

B.2 INSTRUCTPIX2PIX

  • InstructPix2Pix uses a source image and text instruction, but lacks explicit 3D control and struggles with complex multi-object edits.In the described evaluation, it struggles to produce fine object-pose changes specified in text.Source: §B.2 · p. 18

B.3 T2I ADAPTERS

  • T2I-Adapters incorporate spatial signals such as depth maps or segmentation masks into pre-trained diffusion models; the evaluation tests source-image reconstruction.The described setup conditions on spatial modalities corresponding to the source image.Source: §B.3 · p. 18

B.4 NEURAL ASSETS

  • Neural Assets represents appearance and pose separately and supports object translation, rotation, and rescaling.The evaluation conditions on source modalities and target poses, then measures reconstruction loss against the target image.Source: §B.4 · p. 18

B.5 CONTROLNET

  • ControlNet adds trainable layers for spatial signals but does not support object-centric editing, since conditioning changes can alter the whole image.The described evaluation conditions on source-image modalities and measures reconstruction performance.Source: §B.5 · p. 18

C HYPERPARAMETERS

This section identifies the implementation and training hyperparameters for Neural USD and presents them in Table 1.

C HYPERPARAMETERS

D QUALITATIVE BASELINES

Neural USD, 3DIT, and InstructPix2Pix are compared for object-pose conditioning across four datasets. Neural USD satisfies target poses while preserving appearance; InstructPix2Pix does not elicit object movement.

D QUALITATIVE BASELINES

  • Across MOVi-E, Objectron, Waymo Open, and EgoTracks, the comparison conditions models on source images and target 3D bounding boxes or textual prompts.Neural USD and 3DIT use 3D bounding-box targets; InstructPix2Pix uses textual prompts.Source: Figure 9 · p. 20
  • Neural USD satisfies the desired pose while preserving foreground and background appearance.Source: Figure 9 · p. 20
  • InstructPix2Pix fails to elicit object movement.Source: Figure 9 · p. 20

E ITERATIVE WORKFLOW EXAMPLES

The examples show iterative edits to object pose, appearance, and geometry, followed by background changes, including edits to a chair and a laptop.

E ITERATIVE WORKFLOW EXAMPLES

  • For the chair example, the sequence changes pose, then object appearance and geometry, and finally the background, using different aspects of one image for geometry and background conditioning.Source: Figure 10 · p. 21
  • The panels are labeled Original, Pose, Appearance, Geometry, and Background.Source: E ITERATIVE WORKFLOW EXAMPLES · p. 21
  • The laptop example replaces the object and also demonstrates editing its pose and the background.Source: Figure 11 · p. 21

F ADDITIONAL EXPERIMENTAL RESULTS

Additional results show pose-control examples across datasets, a limited-data generalization boundary, and background replacement as an asset swap.

F ADDITIONAL EXPERIMENTAL RESULTS

  • Additional 3D pose-control examples are shown for Objectron and MOVi-E, with 2D bounding-box control examples for EgoTracks.Source: Figure 12 · p. 22, Figure 13 · p. 22, Figure 14 · p. 23
  • With only a handful of labeled datasets totaling fewer than 50,000 sequences, Neural USD fails to generalize to new object categories.The caption says this lack of generalization can likely be remedied by co-training on more readily available 2D bounding-box datasets.Source: Figure 15 · p. 23
  • Neural USD treats the background as another asset that can be swapped using reference modalities.Source: Figure 16 · p. 24

Related papers

Loading 2510.23956v1…