PaperlayerView exact revision on arXiv

Source-linked AI summary

Edit4D: Editable 4D Scene Generation with Textual Control

Chanwoo Kim, Tackgeun You, Hwasup Lim

arXiv:2610.21668v1cs.CV

TL;DR

Holistic 4D scene generation couples backgrounds and dynamic characters, limiting object-level editing and flexible camera control. Edit4D generates these components separately and composes them through a staged pipeline. The authors report state-of-the-art generation quality, improved preservation of unedited content and editing time, and robustness to large camera viewpoint changes.

  • Problem

    Holistic 4D scene generation couples backgrounds and dynamic characters, making object-level editing and flexible camera control difficult.

    Source: §1 · p. 2

  • Method

    Edit4D generates separate background and dynamic-character representations, then composes them through stage-specific prompts, text-guided placement, and contact-aware optimization.

    Source: §1 · p. 2

  • Results

    Edit4D achieves state-of-the-art generation quality on VBench, better preserves unedited scene content and reduces editing time versus holistic regeneration, and remains robust under large viewpoint changes.

    Source: §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The component-based framework supports object-level control, composition, and dynamic camera control in editable 4D scenes.

    Source: §1 · p. 2, §5 · p. 9

    Errors from pretrained intermediate modules can propagate to the final scene, and severe upstream errors may remain despite refinement.

    Source: H. Limitations · p. 21

Abstract

from arXiv · show

We introduce Edit4D, an editable text-to-4D scene generation framework that constructs 4D scenes from separate background and dynamic character representations. Recent advances in generative models have enabled text-driven dynamic scene synthesis, but editable 4D scene generation remains challenging due to the need to preserve spatial geometry and temporal dynamics. Existing approaches typically generate holistic scene videos, where the background and dynamic characters are coupled, making object-level editing and flexible camera control difficult. Edit4D addresses these limitations through a stage-specific generation pipeline. Given a user instruction, Edit4D first decomposes it into stage-specific prompts for background, dynamic character, and scene-character composition. Using these prompts, we synthesize multi-view scene images with text-to-image and camera-conditioned generative priors and reconstruct a background 3D Gaussian splatting representation. We then generate a character image, animate it with an image-conditioned video prior, reconstruct a 4D mesh, and bind Gaussian primitives to obtain a character representation. Finally, both components are composed through text-guided segmentation and contact-aware optimization. A global scene refinement is then performed to improve harmonization and rendering consistency. By integrating stage-specific components, Edit4D enables controllable camera motion, flexible object-level editing, and the practical construction of editable 4D scenes. Project page: https://chanwo0kim.github.io/Edit4D/

1. Introduction

Edit4D addresses the difficulty of editing holistic 4D scenes by generating separate background and dynamic-character representations for composition. Its stage-specific pipeline supports component-level control, while experiments report state-of-the-art performance and controllable camera motion.

1. Introduction

  • Existing object-level methods generate dynamic subjects without complete backgrounds, while holistic scene-level methods make object-level editing difficult.Limited generated viewpoints can also degrade novel-view rendering, and localized errors may require costly sequence regeneration.Source: §1 · p. 2
  • Edit4D decomposes user instructions into stage-specific prompts and generates background and character components separately before composition.The background is reconstructed as a 3D Gaussian representation; the animated character is reconstructed as a 4D mesh with bound Gaussian primitives.Source: §1 · p. 2
  • Edit4D reports state-of-the-art performance against recent 4D generation methods while supporting controllable camera motion and fine-grained object-level editing.Its contributions also describe preserving high perceptual quality.Source: §1 · p. 2
  • Text-guided masks localize the character placement region, and contact- and penetration-aware constraints optimize its position.Global scene refinement improves appearance consistency and rendering details.Source: §1 · p. 2

2. Related Works

Related work spans text- and video-driven 3D/4D generation, scene representations, and scene composition. Edit4D differs by composing separate scene components and automatically placing dynamic characters in generated backgrounds.

2.1. 3D and 4D Scene Generation

  • Object-level 4D methods generate dynamic objects, whereas scene-level methods generate complete scenes; both typically use holistic synthesis that limits later component control.Edit4D instead uses separate representations to support fine-grained modification while preserving spatio-temporal coherence.Source: §2.1 · p. 3

2.2. Scene Representation

  • Gaussian Splatting uses rasterized Gaussian primitives for real-time rendering, while dynamic 4D methods jointly model spatial structure and motion.Surface-aligned primitives and mesh-attached Gaussians address geometry-recovery challenges under photometric supervision.Source: §2.2 · p. 3

2.3. Scene Composition

  • Prior scene-composition methods use masks to localize editable regions, while human-scene placement methods use contact or penetration constraints for plausible interactions.Edit4D applies text-guided segmentation and contact-aware optimization to place dynamic characters in generated backgrounds.Source: §2.3 · p. 3

3. Method

Edit4D builds editable 4D scenes by composing separate background and dynamic-character representations through a stage-specific pipeline. The representation supports component-level editing and camera control, while generation proceeds from instruction decomposition through composition and global refinement.

3.1. Editable 4D Scene with Gaussian Splatting

  • Separate background and time-varying object Gaussians are concatenated into a scene representation, supporting object-level editing and rendering from arbitrary camera poses.The component-wise design supports changing individual objects’ appearance, motion, or placement.Source: §3.1 · p. 4

3.2. Stage 1: Instruction Decomposition

  • A large language model decomposes one user instruction into stage-specific prompts for background generation, character generation, and scene–character composition.The prompts preserve the original semantics while supplying each module with the information it requires.Source: §3.2 · pp. 4–5

3.3. Stage 2: Background Scene Generation

  • Text-to-image synthesis supplies a background reference, camera-conditioned generation expands it into geometrically consistent views, and those views are reconstructed as a background Gaussian representation.The views provide spatial observations and parallax cues for reconstruction.Source: §3.3 · p. 5

3.4. Stage 2: Dynamic 3D Character Generation

  • An image-conditioned video prior animates a generated character image, which is reconstructed as a deformable mesh sequence and converted into a 4D Gaussian representation by binding Gaussians to mesh faces.A learned deformation field predicts per-time Gaussian attribute offsets to address fine-grained deformation that mesh binding alone may not capture.Source: §3.4 · p. 5

3.5. Stage 3: Scene–Character Composition

  • Text-guided segmentation and depth filtering identify a target 3D region; bounding-box initialization and contact and penetration losses guide character placement and global transform optimization.The initialization places the character near the target surface to improve optimization stability and reduce failures from implausible placements.Source: §3.5 · pp. 5–6

3.6. Global Scene Refinement

  • Global refinement uses a text-conditioned diffusion prior to jointly optimize the background, canonical character Gaussians, and deformation field for appearance harmonization and visual detail.It samples background-generation cameras at canonical time and uses frontal views for dynamics to support multi-view consistency, character identity, and temporal consistency.Source: §3.6 · p. 6

4. Experiments

Edit4D achieves strong generation and editing results while producing coherent scenes under camera changes. Its component-based representation preserves unedited content, and ablations support the roles of composition constraints and global refinement.

4.1. Evaluation Metrics

  • Evaluation uses frontal-view videos and VBench’s Text Align, Consistency, and Aesthetic metrics to assess prompt similarity, temporal coherence, and visual quality.Consistency uses DINO and CLIP feature similarities; Aesthetic uses the LAION Aesthetic Predictor.Source: §4.1 · p. 6

4.2. Quantitative Results

  • Edit4D achieves state-of-the-art VBench performance across three settings, and 25 participants preferred it over Free4D for both 3D structure and visual quality.The user study used the same video generation model for both methods.Source: §4.2 · p. 6

4.3. Qualitative Results

  • Edit4D produces coherent, natural scenes with quality comparable to prior methods and maintains more stable visual quality than Free4D under large camera motions.Figure 4 compares 4Real, Dream-in-4D, 4Dfy, and Free4D across three settings; Figure 5 shows frontal and novel views over time.Source: §4.3 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8

4.4. Evaluation on Editability

  • Edit4D achieves higher editing success and background preservation while reducing editing time compared with full-scene regeneration.Free4D’s holistic representation requires full regeneration, which can change background content such as the bookshelf and rug.Source: §4.4 · p. 7, Figure 6 · p. 8

4.5. Ablation Study

  • Removing initialization, contact loss, penetration loss, or global refinement degrades metrics; the full model achieves the best quantitative results.Without initialization, placement can be incorrect; without contact loss characters float, and without penetration loss they enter scene geometry. Global refinement improves fine details and visual coherence.Source: §4.5 · p. 8, Table 4 · p. 8, Figure 7 · p. 8

5. Conclusion

Edit4D separates background scenes and dynamic characters for editable 4D generation. Evaluations report strong generation quality, better preservation of unedited content with less editing time than holistic regeneration, and robustness to large camera viewpoint changes.

5. Conclusion

  • Edit4D separates background scenes and dynamic characters to support object-level control and composition in editable 4D scene generation.The paper reports state-of-the-art VBench generation quality, better preservation of unedited content, reduced editing time, coherent scenes under large viewpoint changes, and text-guided composition with contact-aware optimization.Source: §5 · p. 9

A. Implementation details

Edit4D uses modular stages to generate a background and dynamic character, compose them, and refine the resulting scene. Its implementation separates stage-specific prompts and representations, with reusable assets supporting iterative editing.

A. Implementation details

  • The modular stages can use alternative models with compatible inputs and outputs without changing the overall pipeline.The implementation decomposes instructions, generates the background and character, composes them, and performs global refinement.Source: A. Implementation details · p. 13

A.1. Instruction Decomposition

  • A single user instruction is decomposed into separate prompts for background, character, motion, and composition.Each prompt contains only information required by its stage.Source: §A.1 · p. 13

A.2. Background Scene Generation

  • A frontal background image is extended to geometrically consistent multi-view images, which are used to reconstruct a background Gaussian splatting representation.The implementation uses VistaDream to generate trajectory-conditioned multi-view inpainting images and reconstruct the background in a unified pipeline.Source: §A.2 · p. 13

A.3. Dynamic 3D Character Generation

  • A character image is animated into a 60-frame video, then reconstructed as a temporally consistent mesh sequence and converted to dynamic Gaussians bound to mesh faces.The implementation renders from 30 cameras and adds a lightweight deformation field to improve temporal stability under non-rigid deformation.Source: §A.3 · p. 13

A.4. Scene–Character Composition

  • Character composition optimizes a global rotation and translation, initialized from character and target bounding boxes, with contact and penetration losses over 1,000 iterations.The character is initialized slightly above the support surface; a signed distance field penalizes scene points inside the character mesh.Source: §A.4 · p. 13

A.5. Global Scene Refinement

  • Global refinement optimizes the background, canonical character Gaussians, and deformation network for 4,000 iterations, using multi-view canonical frames followed by canonical or frontal dynamic frames.For shadows, per-view HDR lighting estimates are aligned and median-aggregated; character opacity along sampled light directions is used to estimate blocked light, and the illumination loss is applied only to the background contribution.Source: §A.5 · p. 14

A.6. Runtime

  • Edit4D takes approximately 2 hours per example on one RTX Pro 6000 Blackwell GPU, including 70 minutes for character generation and 45 minutes for global refinement.For edits, it can reuse background and character assets, requiring only composition and optional refinement rather than full-scene regeneration.Source: §A.6 · pp. 14–15

B. Baseline and Datasets

The evaluation uses 13 unique instructions across three settings and compares Edit4D with object-level and general text-to-4D baselines. A randomized, anonymous user study also compares original-scene structure and visual quality, and background preservation after edits.

B. Baseline and Datasets

  • The evaluation compares Edit4D with two object-centric and two general text-to-4D baselines across three settings, using 13 unique instructions.The instructions were manually constructed to match public baseline visuals because original prompts were unavailable for most baselines.Source: B. Baseline and Datasets · p. 15

C. Capability

  • Edit4D represents background and dynamic characters as separate Gaussian representations, supporting independent component editing and flexible scene–character composition.The capability comparison considers input modality, output representation, separation, component editing, and character composition.Source: C. Capability · p. 15

D. Details on User Study

  • In an anonymous pairwise study, 25 participants compared Edit4D and Free4D on 13 original scenes and five scenes for each of two editing tasks.Participants assessed 3D structure and visual quality for original scenes, then background preservation after recomposition or character appearance edits.Source: D. Details on User Study · p. 15

E. Evaluation on Editability

Editability is evaluated by comparing Edit4D with Free4D on background preservation after character appearance edits and scene–character recomposition. Edit4D achieves higher editing success and preservation while requiring less editing time.

E. Evaluation on Editability

  • Free4D represents scenes holistically, so each edit requires regeneration of the full 4D scene.The comparison uses Free4D’s publicly released implementation and settings, with edited reference videos generated using the same video model as Edit4D.Source: E. Evaluation on Editability · p. 17
  • The evaluation tests five character scenes for each editing task across multiple viewpoints and time steps, measuring background consistency outside the edited region.A shared mask covers the union of the original and edited character regions to exclude changes to character position, scale, or appearance.Source: E. Evaluation on Editability · p. 17
  • Edit4D consistently achieves higher editing success and background preservation than Free4D on both editing tasks, while requiring substantially less editing time.Success is assessed by three VLMs; preservation is measured with CLIP and DINOv2 similarities on masked original and edited images.Source: E. Evaluation on Editability · p. 17

F. Additional qualitative results

Additional results illustrate appearance editing, asset insertion, and scene–character recomposition, alongside character-placement variations and novel-view renderings. The examples show edits that preserve remaining scene content and compositions spanning different contact configurations.

F. Additional qualitative results

  • Edit4D reuses existing scene assets and better preserves unedited background content across frontal and rotated views than Free4D, which requires full scene regeneration for each edit.The comparison covers scene–character recomposition and character appearance editing.Source: Figure 10 · p. 18
  • Edit4D supports character appearance modification, asset insertion, and scene–character recomposition without regenerating the entire scene.Examples include placing a hat asset on a character, repositioning it to a specified sofa region, and changing color values within a 3D cloth mask.Source: F. Additional qualitative results · pp. 17, 19, Figure 11 · p. 19
  • The composition framework integrates Animatable Gaussians for plausible multicharacter compositions and supports non-contact floating characters using a distance-thresholded objective.The main pipeline uses a video-to-mesh model that does not require explicit rigging or skeleton annotations.Source: F. Additional qualitative results · p. 19
  • Additional results show coherent, visually plausible scenes across diverse prompts and high visual quality under large camera viewpoint changes.Source: F. Additional qualitative results · p. 19

G. Analysis on Contact Map

Contact-map analysis indicates that text-guided placement can align high-contact regions with the specified target and concentrate contact on plausible character support regions. Supplementary figures show additional comparisons, novel views, and an occlusion-related composition failure.

G. Analysis on Contact Map

  • High-contact regions align with the text-specified target, while contact responses concentrate around plausible character support regions.The analysis presents this as evidence of plausible scene–character composition while maintaining editability and rendering quality.Source: G. Analysis on Contact Map · p. 19
  • Figure 13 provides additional qualitative comparisons with 4Real [75], Dream-in-4D [79], 4Dfy [2], and Free4D [35] across three experimental settings.Source: Figure 13 · p. 20
  • Figure 14 visualizes generated 4D scenes from the frontal view and four novel camera views over time.Source: Figure 14 · p. 20
  • Figure 15 encodes the scene object in black, the character in blue, and higher contact intensity in brighter yellow, alongside the target object mask.Source: Figure 15 · p. 21
  • Figure 16 shows that partial chair occlusion can produce an incomplete target region, inaccurate character placement, and desk intersection in rotated views.Source: Figure 16 · p. 21

H. Limitations

The limitations concern target visibility for text-guided composition and errors from pretrained intermediate modules. Future work proposes broader editable assets, interactive composition control, and improved robustness.

H. Limitations

  • Partially occluded targets can yield incomplete regions, inaccurate character placement, and intersections with surrounding geometry.The representative failure involves a chair occluded by a desk; stronger 3D scene understanding or occlusion-aware target completion could alleviate it.Source: H. Limitations · p. 21
  • Errors in pretrained intermediate predictions can propagate, and severe upstream errors may remain despite refinement of moderate inconsistencies.Examples include character-reconstruction inconsistencies, segmentation-based localization errors, and artifacts in generated images or reference videos.Source: H. Limitations · p. 21

I. Future Work

  • Future work proposes more reusable editable assets, interactive control of placement and contact, better occlusion handling, and reduced sensitivity to intermediate errors.Source: §I · p. 21

Related papers

Loading 2610.21668v1…