PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models

Ziyi Wu, Yulia Rubanova, Rishabh Kabra, Drew A. Hudson, Igor Gilitschenski, Yusuf Aytar, Sjoerd van Steenkiste, Kelsey R. Allen, Thomas Kipf

arXiv:2406.09292v2cs.CVcs.AIcs.LG

TL;DR

Text alone gives limited precision for image generation, and existing approaches do not provide intuitive 3D control of multiple objects in real-world scenes. The paper trains Neural Assets from video frames to separate appearance from pose and conditions a fine-tuned image generator on them. It reports state-of-the-art 3D-aware editing results across synthetic and real-world scenes, with cross-scene asset transfer.

  • Problem

    Text alone is often insufficient for precise image control, and existing approaches have limited 3D understanding or struggle with multi-object real-world scenes.

    Source: §1 · p. 1

  • Method

    Neural Assets encode object appearance from one video frame and reconstruct it in another using target-frame 3D bounding boxes, separating appearance from pose for a fine-tuned image generator.

    Source: §1 · p. 2

  • Results

    The paper reports state-of-the-art 3D-aware object editing on synthetic and real-world scenes, with object and background transfer across scenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors frame Neural Assets as an important step toward general-purpose neural-based simulators.

    Source: §5 · p. 10

    The current method controls rigid 3D pose and background composition, but not structural changes such as deformation, articulation, or decomposition.

    Source: §5 · p. 10

Abstract

from arXiv · show

We address the problem of multi-object 3D pose control in image diffusion models. Instead of conditioning on a sequence of text tokens, we propose to use a set of per-object representations, Neural Assets, to control the 3D pose of individual objects in a scene. Neural Assets are obtained by pooling visual representations of objects from a reference image, such as a frame in a video, and are trained to reconstruct the respective objects in a different image, e.g., a later frame in the video. Importantly, we encode object visuals from the reference image while conditioning on object poses from the target frame. This enables learning disentangled appearance and pose features. Combining visual and 3D pose representations in a sequence-of-tokens format allows us to keep the text-to-image architecture of existing models, with Neural Assets in place of text tokens. By fine-tuning a pre-trained text-to-image diffusion model with this information, our approach enables fine-grained 3D pose and placement control of individual objects in a scene. We further demonstrate that Neural Assets can be transferred and recomposed across different scenes. Our model achieves state-of-the-art multi-object editing results on both synthetic 3D scene datasets, as well as two real-world video datasets (Objectron, Waymo Open).

1 Introduction

Text-to-image diffusion models can generate realistic scenes, but text alone offers limited precision and existing controls do not provide intuitive 3D control of multiple objects in real-world scenes. Neural Assets use video-derived training signals to represent object appearance separately from pose, supporting 3D editing and compositional generation.

1 Introduction

  • Existing conditioning methods lack intuitive 3D object control, while methods introducing 3D geometry struggle with multi-object real-world scenes because scalable paired images and 3D annotations are hard to obtain.Source: §1 · p. 1
  • Video frames provide naturally occurring 3D pose changes, such as bounding-box changes, as training labels for multi-object editing.Source: §1 · p. 2
  • Neural Assets encode each object's consistent appearance with variable 3D pose, trained by reconstructing its appearance in another video frame using that frame's 3D bounding boxes.Source: §1 · p. 2
  • The paper reports state-of-the-art 3D-aware object editing on synthetic and real-world scenes and supports transferring objects and backgrounds across images.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work explores spatial control, 3D-aware generation, personalization, and object-centric representations, but the paper identifies limits in 3D control, real-world scenes, and generalizable object representations. It positions Neural Assets as an encoder-based approach connected to a pre-trained diffusion model for multi-object 3D editing.

2 Related Work

  • 2D diffusion controls use prompts, attention, latents, or fine-tuned spatial inputs, while this work uses 3D bounding boxes for rotation and occlusion-aware control.Source: §2 · p. 2
  • Prior 3D-aware diffusion methods trained on multi-view assets focus on single objects without backgrounds [105] [61], while real-world multi-object methods have other constraints [67] [8].Source: §2 · p. 3
  • Personalized generation often handles one subject without spatial control, and Subject-Diffusion [65] uses 2D boxes but cannot explicitly control 3D pose.Source: §2 · p. 3
  • Object-centric slots support editing attributes and poses [93] [87] [46], but the paper distinguishes Neural Assets by connecting fine-tuned visual encoders to pre-trained diffusion models for complex real-world scenes.Source: §2 · p. 3

3 Method: Neural Assets

Neural Assets adapt the appearance-and-pose structure of 3D assets to image diffusion models, using paired observations to learn object representations that support 3D-aware editing and scene composition.

3.1 Background: 3D Assets in Computer Graphics

  • A 3D asset is represented as appearance descriptors and pose, such as canonical shape, textures, rigid transformation, and scaling.Source: §3.1 · p. 3

3.2 Neural Assets

  • A Neural Asset factorizes a learnable object representation into appearance and pose, enabling their independent control without requiring explicit 3D object mapping beyond 3D pose.The representation is used for 3D-aware object editing and compositional generation.Source: §3.2 · pp. 3–4, §3 · p. 3

3.2.1 Asset Encoding

  • Appearance tokens pool box-selected visual features in one encoder pass, while pose tokens encode 3D-box information; the approach assumes 3D annotations, whose high-quality acquisition at scale remains open.The pose representation uses projected box corners and depth, embedded through an MLP.Source: §3.2.1 · pp. 4–5
  • Appearance and pose tokens are concatenated and linearly projected into Neural Asset tokens, which can replace or append to text tokens; a separate background token supports scene control.The background is encoded after masking foreground boxes and can support scene swapping or control of global properties such as lighting.Source: §3.2.1 · p. 5

3.2.2 Generative Decoder

  • A token-conditioned generative model is fine-tuned end-to-end from a pretrained text-to-image checkpoint to reconstruct images from Neural Assets.The framework assumes the generator accepts a sequence of conditioning tokens; Stable Diffusion v2.1 is the representative decoder.Source: §3.2.2 · p. 5

3.3 Learning and Inference

  • Paired video frames train source-frame appearance with target-frame 3D-box pose to reconstruct the target; at inference, boxes control object movement, rotation, and scale, while assets and backgrounds can be recomposed or swapped.The paper also describes rotating an object's pose and replacing it with an object from another image.Source: Figure 2 · p. 4, §3.3 · pp. 5–6

4 Experiments

Experiments evaluate Neural Assets on synthetic and real-world datasets, finding strong single- and multi-object editing results alongside compositional scene control. Ablations examine the visual encoder, background modeling, and paired-frame training.

4.1 Experimental Setup

  • Experiments span four datasets, from synthetic scenes with limited object motion to real-world videos with complex backgrounds and camera movement.OBJect has a static camera; MOVi-E includes linear camera motion and multiple moving objects, while Objectron and Waymo Open use real-world video.Source: §4.1 · p. 6
  • Baselines are applied sequentially for multi-object edits, and evaluation combines image-level quality metrics with object-level fidelity measures.The evaluation includes PSNR, SSIM, LPIPS, FID, and DINO feature similarity; experiments use fixed source and target frames across runs.Source: §4.1 · pp. 6–7
  • Experiments use 256 × 256 images, a jointly fine-tuned DINO ViT-B/8 encoder, and 50-step DDIM inference.Training uses Adam with batch size 1536 on 256 TPUv4 chips.Source: §4.1 · p. 7

4.2 Main Results

  • The model achieves state-of-the-art performance on three single-object tasks and outperforms baselines across MOVi-E, Objectron, and Waymo Open.Qualitative comparisons favor it in object identity preservation, editing accuracy, and background modeling.Source: Figure 5 · p. 7, §4.2 · p. 8

4.3 Controllable Scene Generation

  • On Waymo Open, the model supports car translation and rotation, object removal and transfer across scenes, and background swapping with environmental adaptation.Examples include consistent novel views during rotation, occlusion handling, and car lights gaining specular highlights in a night scene.Source: §4.3 · pp. 8–9, Figure 6 · p. 8, Figure 7 · p. 8

4.4 Ablation Study

  • MAE and DINO outperform CLIP in the multi-object setting, with DINO and joint encoder fine-tuning producing the best performance.The authors attribute the gains to finer-grained features and richer 3D information than CLIP’s high-level image semantics.Source: §4.4 · p. 9
  • Background appearance and relative camera pose together perform best in image-level metrics and also improve object-level metrics.Providing background appearance and pose means object tokens need not infer them, yielding more disentangled Neural Assets.Source: §4.4 · pp. 9–10
  • Paired-frame training outperforms single-frame training and the no-positional-encoding variant; remaining failures include symmetry ambiguity and camera-object motion entanglement.Different-frame reconstruction encourages inference of underlying 3D structure and realistic lighting rather than copying pixels; the authors suggest more diverse training data may resolve the failures.Source: §4.4 · p. 10, Figure 10 · p. 10

5 Conclusion

The conclusion presents Neural Assets as vector-based representations that disentangle appearance and pose for controllable 3D scene generation and cross-scene transfer. It also identifies failure cases and scope boundaries involving symmetry, motion, structural changes, and 3D-box annotations.

5 Conclusion

  • Neural Assets support multi-object 3D control and asset transfer across synthetic and real-world scenes.The authors describe the work as an important step toward general-purpose neural-based simulators.Source: §5 · p. 10
  • Symmetry ambiguity can flip a cup handle after a 180-degree rotation, while Objectron’s camera-only motion can entangle global camera motion with local object movement.The authors suggest larger datasets with more diverse object and camera motion may resolve both issues.Source: §5 · p. 10
  • Current control covers rigid 3D pose and background composition, not structural changes such as deformation, articulation, or object decomposition.The authors say adaptation to other changes depends on suitable datasets.Source: §5 · p. 10
  • The approach is currently limited to datasets with 3D bounding-box annotations.The authors point to scalable 3D annotation pipelines as a possible route to larger open-domain training datasets.Source: §5 · p. 10

A Detailed Experimental Setup

This section provides the experimental details needed to interpret the study’s evaluation and implementation.

A Detailed Experimental Setup

A.1 Datasets

The setup spans synthetic and real-world datasets with varying object and camera motion, and describes two editing baselines. Dataset-specific processing and pose conventions shape the evaluations.

A.1 Datasets

  • OBJect uses Blender-rendered scenes with up to four objects and one object edited per example; MOVi-E has 11–23 objects, collisions, and linear camera motion.OBJect translations and rotations remain on the ground; MOVi-E uses real-world scanned objects.Source: §A.1 · p. 18
  • Objectron provides 15,000 object-centric clips across nine categories, while Waymo Open contributes 1,000 self-driving videos and uses front-view car annotations.For Waymo, only yaw is annotated; the other two rotation angles are treated as zero, and projected 3D boxes provide associated 2D boxes because the supplied 2D and 3D boxes are misaligned.Source: §A.1 · p. 18
  • Images are resized to 256 × 256; bike videos are discarded from Objectron, small Waymo cars are filtered, and augmentation is applied only to Waymo Open.Source: §A.1 · p. 18

A.2 Baselines

  • 3DIT fine-tunes Zero-1-to-3 for scene-level edits using instructions derived from target object poses, but does not support large viewpoint changes because it lacks camera-pose encoding.Source: §A.2 · p. 18
  • Chained performs object editing by removing and inpainting the original region, synthesizing the object in its new pose, then placing it at the target location.The pipeline uses SAM for segmentation, Stable Diffusion v2 inpainting, Zero-1-to-3 for re-posing, and alpha blending for placement; Zero-1-to-3 does not support camera roll rotation.Source: §A.2 · pp. 18–19

A.3 Evaluation Settings

Evaluation measures edited-image accuracy at both whole-image and object-box levels, and separately measures object identity preservation.

A.3 Evaluation Settings

  • PSNR, SSIM, LPIPS, and FID measure edited-image accuracy across the entire image and within edited-object bounding boxes.For box-level metrics, each object is cropped and evaluated without resizing.Source: §A.3 · p. 19
  • Object identity preservation is evaluated using DINO feature similarity, computed as cosine similarity between predicted and ground-truth cropped-object features.Source: §A.3 · p. 19

A.4 Our Implementation Details

The implementation uses DINO visual features and projected 3D pose tokens to condition a fine-tuned Stable Diffusion model. Joint training and classifier-free guidance are used to train and run the framework.

A.4 Our Implementation Details

  • DINO ViT-B/8 features form four-token object appearances, while MLPs project 3D boxes and relative camera poses into Stable Diffusion’s 1024-dimensional conditioning space.Background features come from images with object-box pixels masked to 0.5.Source: §A.4 · p. 19
  • All components are trained jointly, with 200k steps for OBJect and MOVi-E and 50k for Objectron and Waymo Open.The paper reports 24 hours for the first pair and 6 hours for the latter pair; classifier-free guidance drops appearance and pose tokens with 10% probability.Source: §A.4 · p. 19
  • At inference, the model generates images with 50-step DDIM sampling and a classifier-free guidance scale of 2.0.The authors report that scales from 1.5 to 4 work well.Source: §A.4 · p. 19
  • Single-object OBJect results cover translation, rotation, and removal on seen and unseen object subsets, while multi-object results cover MOVi-E, Objectron, and Waymo Open.Metrics are computed within edited object boxes for the single-object table and on whole images and object boxes for the multi-object table; results are averaged over three seeds.Source: Table 1 · p. 20, Table 2 · p. 20

B.1 Full Benchmark Results

The appendix reports full quantitative results for OBJect, MOVi-E, Objectron, and Waymo Open, including additional metrics and evaluation scopes. The reported trends are similar to the main paper, with Neural Assets significantly outperforming baselines across datasets.

B.1 Full Benchmark Results

  • Neural Assets significantly outperform baselines across OBJect, MOVi-E, Objectron, and Waymo Open, with trends similar to the main paper.The appendix adds FID and unseen-object results for OBJect, plus FID, DINO feature similarity, and image-level results for the other three datasets.Source: §B.1 · p. 20
  • The appendix reports full quantitative results for OBJect in Table 1 and for MOVi-E, Objectron, and Waymo Open in Table 2.Table 1 covers seen and unseen objects; Table 2 reports whole-image and object-box metrics.Source: §B.1 · p. 20
  • Additional evaluation includes FID and unseen-object results on OBJect, and FID, DINO feature similarity, and image-level results on the other datasets.The reported additional metrics and scopes are compared with the main paper’s results.Source: §B.1 · p. 20

B.2 Full Ablation Results

The appendix reports Objectron ablations and demonstrates controllable scene generation. Across the showcased scenarios, Neural Assets separate foreground-object control from background modeling, including background inpainting and shadow rendering.

B.2 Full Ablation Results

  • Objectron ablations show similar trends across metrics at both image and object levels.The appendix presents the studies in Tables 3, 4, and 5.Source: §B.2 · p. 20

B.3 Controllable Scene Generation

  • On Objectron, Neural Assets can rotate foreground objects while keeping the background fixed or swap backgrounds between scenes, despite videos containing only camera movement and static objects.The model inpaints masked background regions not occupied by the novel object and renders realistic shadows around it.Source: §B.3 · pp. 21–22, Figure 11 · p. 21

B.4 Ablation on 3D Pose Representations

The appendix describes a projected-corner pose representation and its advantage for learning car rotation when rotated training examples are scarce. It also outlines the Stable Diffusion conditioning setup, broader impacts, and funding disclosure.

B.4 Ablation on 3D Pose Representations

  • The pose vector concatenates the image-plane coordinates and depth values of a 3D box’s four projected corners, encoding position, rotation, scale, and occlusion cues.The projected points resemble a local coordinate frame for the object.Source: §B.4 · p. 22, Figure 14 · p. 24
  • Projected local-coordinate pose features provide more rotation training signal for mostly unturned Waymo cars than directly inputting rotation angles, which the model tends to ignore.Alternative center, size, and rotation representations perform similarly on MOVi-E and Objectron, but have significantly worse learned rotation controllability on Waymo.Source: §B.4 · p. 22

C Background on Stable Diffusion

  • Stable Diffusion iteratively denoises latent image codes, and this work replaces its text embeddings with Neural Assets for appearance and pose control.The latent codes are produced by a VAE tokenizer, and the denoiser is conditioned on text in the original text-to-image setup.Source: C Background on Stable Diffusion · p. 22

D Broader Impacts

  • Neural Assets provide an interface for controlling generative models, with potential applications in computer graphics and robotics.The paper describes this as a potential benefit and says the interface may support more interpretable AI algorithms.Source: D Broader Impacts · p. 24
  • The pipeline inherits dataset-selection bias from its pretrained generative models; the current approach controls rigid objects but does not treat humans as assets.The authors say further study is needed on how this bias affects performance.Source: D Broader Impacts · p. 24

E Funding Disclosure

Loading 2406.09292v2…