Instructive3D: Editing Large Reconstruction Models with Text Instructions

1. Introduction

• The work targets fine-grained editing control that existing LRMs lack, for applications including augmented reality, animation, and gaming.Source: §1 · pp. 1–2
• The method integrates fine-grained text-prompt editing into existing LRM architectures by conditioning triplane features with latent diffusion.Source: §1 · p. 2
• The proposed data-curation pipeline addresses the difficulty of acquiring edited 3D object pairs by leveraging existing foundational models.The introduction describes using InstructPix2Pix to create edited images, then a pretrained LRM to obtain triplane latents for diffusion supervision.Source: §1 · p. 2
• Instructive3D produces superior-quality 3D objects over baselines with various editing prompts on standard datasets such as Objaverse.Source: §1 · p. 2

2.1. Diffusion Models

• Diffusion models add Gaussian noise over timesteps and train a conditioned denoiser to iteratively recover the clean sample.Source: §2.1 · p. 2
• Pretrained 2D diffusion models support 3D generation methods such as DreamFusion [35], DreamGaussian, and GaussianDreamer, with differing representations and processing pipelines.Source: §2.1 · p. 2
• Score Jacobian Chaining [50] aggregates diffusion-model scores across camera viewpoints, while Zero1to3 [25] changes an object's viewpoint from a single RGB image but uses slow per-instance optimization.Source: §2.1 · p. 3

2.2. 3D Reconstruction

• Single-view reconstruction spans representations including voxels, meshes, radiance fields, SDFs, and 3D Gaussians, but consistency in generated geometry and colors remains challenging.Source: §2.2 · p. 3
• LRMs use feed-forward architectures to map 2D information directly to triplanes for fast, generalizable 3D reconstruction, avoiding test-time optimization.Source: §2.2 · p. 3
• Other reconstruction approaches differ in representation: CRM uses multiview diffusion to generate coordinate maps, while GRM builds on 3D Gaussian splatting.Source: §2.2 · p. 3

3.1. Training Procedure

• Training learns only the triplane diffusion adapter, reusing Real3D to encode input images and decode edited triplane representations.Tri-VAE first compresses and reconstructs triplane features; its weights are then frozen while conditional diffusion learns text-guided edits.Source: §3.1 · p. 4

3.2. Dataset Preparation

• The dataset uses rendered Objaverse LVIS views for Tri-VAE training and manually filtered InstructPix2Pix edits for conditional diffusion.Tri-VAE features are collected from 12,584 objects; edited-image pairs cover 2,108 objects, with identity-prompt examples added to preserve geometry.Source: §3.2 · p. 4

3.3. Tri-VAE: Triplane Variational Auto-Encoder

• Tri-VAE independently encodes each of the three LRM triplane planes, reconstructs them, and concatenates them into a triplane representation.It is trained with reconstruction and KL-divergence losses.Source: §3.3 · p. 5

3.4. LTriD: Latent TriPlane Diffusion Model

• LTriD applies text-conditioned diffusion to stacked Tri-VAE latents, concatenating the original latent with its noisy version to preserve 3D structure.The denoiser is trained first on identity examples and then on paired edited data.Source: §3.4 · p. 5

3.5. Inference

• At inference, an input image is encoded by the pretrained LRM, edited through triplane diffusion, and decoded into a 3D object.Source: §3.5 · p. 5

3.6. Implementation Details

• The implementation compresses each 64 × 64 × 40 plane to 16 × 16 × 12 latents and uses 100 denoising steps at inference.The denoiser has about 870M parameters; Tri-VAE uses about 68M parameters per VAE.Source: §3.6 · p. 5

3. Instructive3D

• Figure 4 shows baseline methods failing the “add a velvet texture to bag” edit, while Instructive3D integrates text editing with 3D generation.The figure compares input prompts and images with Instructive3D and baseline outputs.Source: §3 · p. 3, Figure 4 · p. 6

4.1. Evaluation

• Evaluation uses PSNR, SSIM, LPIPS, FID, KID, and 2D CLIP to assess visual similarity, realism, and text alignment.Objaverse LVIS annotations are used for training and evaluation, with a 45-object synthetic evaluation set.Source: §4.1 · p. 6

4.2. Experimental Results

• Figure 5 compares Real3D outputs with outputs from the 2D VAE and 3D VAE.The first row is Real3D; the next two rows show the 2D VAE and 3D VAE, respectively.Source: Figure 5 · p. 7
• Instructive3D consistently outperforms prior works across LPIPS, PSNR, SSIM, FID, KID, and CLIP, with reported gains in perceptual quality and text alignment.The comparison uses 45 objects with 14 views each; Table 1 presents the quantitative comparison.Source: §4.2 · p. 7, Table 1 · p. 7
• Qualitative comparisons show edits such as added texture or color while preserving original geometry, including across views.The evaluation compares against Text2Mesh, Paint3D, and TEXTure using Real3D meshes and edit prompts.Source: §4.2 · pp. 6–7, §4.1 · p. 6

4.3. Ablation Studies

• The ablation reports superior results for 2D VAEs over a 3D VAE on training loss and input-output triplane MSE, measured on 100 objects.The authors describe separate 2D VAEs for triplane planes as maintaining fidelity while reducing computational complexity.Source: Table 2 · p. 7, §4.3 · p. 7

• Instructive3D integrates text-based prompts into LRM-generated 3D objects through a triplane adapter performing latent diffusion in triplane latent space.The edits are incorporated into generation in a geometrically consistent manner.Source: §5 · p. 7
• The reported qualitative and quantitative results show higher-quality objects that preserve geometry compared with baselines.The method reuses pretrained LRM components and trains only the triplane diffusion adapter.Source: §5 · p. 7

• Implementation details are in Appendix B, and additional experimental results are in Appendix C.Source: A. Introduction · p. 12

• Each Tri-VAE uses three 2D down-encoder and three up-decoder blocks, with three layers per block, 40 input and output channels, four latent channels per plane, and sample size 64.Source: B. Implementation Details · p. 12
• The latent triplane diffusion model uses cross-attention down- and up-blocks, a middle cross-attention block, and two layers per block.Its input and output channels are 24 and 12, its sample size is 16, and CLIP text embeddings feed all cross-attention blocks.Source: B. Implementation Details · p. 12
• The diffusion model's transformer blocks enable both self-attention and cross-attention.Source: B. Implementation Details · p. 12

C. Experimental Results

• Instructive3D preserves geometry and performs edits consistent with input prompts in comparisons with Text2Mesh, Paint3D, and TEXTure.Those systems receive a mesh generated by Real3D and an edit prompt; their outputs are compared with Instructive3D’s generated mesh.Source: C. Experimental Results · p. 12
• The comparison figures place Real3D mesh renderings in the top row and Instructive3D results in the second row.The captions include edits to barrels, cans, bowls, chairs, bags, cups, shoes, and sofas.Source: Figure 10 · p. 16, Figure 14 · p. 20, Figure 17 · p. 23
• The figures cover color and appearance edits as well as added designs, including leaves on a barrel and a velvet texture on a chair.Other prompts request marble texture on a clutch bag and a flame design on a shoe.Source: Figure 12 · p. 18, Figure 21 · p. 27, Figure 23 · p. 29