PaperlayerView exact revision on arXiv

Source-linked AI summary

3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model

Hyun-kyu Ko, Jihyeon Park, Younghyun Kim, Dongheok Park, Eunbyung Park

arXiv:2603.18524v1cs.CV

TL;DR

Existing subject-driven video methods mainly use single-view or textual conditioning, leaving them without sufficient spatial priors for faithful 3D identity across novel views. The paper introduces 3DreamBooth and 3Dapter, using 1-frame optimization and multi-view conditioning to encode and query subject geometry. On 3D-CustomBench, the framework achieves state-of-the-art 3D geometric fidelity and fast convergence.

  • Problem

    Existing single-view and text-driven video customization methods struggle to preserve customized subjects across unseen viewpoints, while multi-view video data are scarce and full-sequence training risks temporal overfitting.

  • Method

    3DreamBooth uses 1-frame optimization to embed a subject’s 3D identity, while 3Dapter adds multi-view visual conditioning through joint optimization with the generation branch.

  • Results

    The framework achieves state-of-the-art 3D geometric fidelity and fast convergence, outperforming existing single-reference baselines and ablations in 3D-aware identity-preserving video generation.

  • Takeaways & Limitations

    The combined approach preserves 3D identity and intricate textures while retaining the pretrained model’s temporal dynamics for customized video generation.

  • Takeaways & Limitations

    Single-token text conditioning remains slow and can lose fine-grained details because the token must learn a complex 3D visual manifold without explicit visual hints.

Abstract

from arXiv · show

Creating dynamic, view-consistent videos of customized subjects is highly sought after for a wide range of emerging applications, including immersive VR/AR, virtual production, and next-generation e-commerce. However, despite rapid progress in subject-driven video generation, existing methods predominantly treat subjects as 2D entities, focusing on transferring identity through single-view visual features or textual prompts. Because real-world subjects are inherently 3D, applying these 2D-centric approaches to 3D object customization reveals a fundamental limitation: they lack the comprehensive spatial priors necessary to reconstruct the 3D geometry. Consequently, when synthesizing novel views, they must rely on generating plausible but arbitrary details for unseen regions, rather than preserving the true 3D identity. Achieving genuine 3D-aware customization remains challenging due to the scarcity of multi-view video datasets. While one might attempt to fine-tune models on limited video sequences, this often leads to temporal overfitting. To resolve these issues, we introduce a novel framework for 3D-aware video customization, comprising 3DreamBooth and 3Dapter. 3DreamBooth decouples spatial geometry from temporal motion through a 1-frame optimization paradigm. By restricting updates to spatial representations, it effectively bakes a robust 3D prior into the model without the need for exhaustive video-based training. To enhance fine-grained textures and accelerate convergence, we incorporate 3Dapter, a visual conditioning module. Following single-view pre-training, 3Dapter undergoes multi-view joint optimization with the main generation branch via an asymmetrical conditioning strategy. This design allows the module to act as a dynamic selective router, querying view-specific geometric hints from a minimal reference set. Project page: https://ko-lani.github.io/3DreamBooth/

1 Introduction

The paper targets 3D-aware customization because single-view or text-driven video methods struggle to preserve subject identity across unseen viewpoints. It introduces 3DreamBooth, 3Dapter, and 3D-CustomBench to address this gap.

  • Motivation: Single-image and text-driven customization methods often produce subjects with rigid 2D appearances that are inconsistent across unseen viewpoints.These approaches lack comprehensive spatial priors for reconstructing the subject’s 3D geometry.
  • Framework: 3DreamBooth uses 1-frame optimization to separate spatial identity from temporal dynamics and embed subject-specific 3D identity without multi-view video datasets.The strategy restricts training to spatial representations, reducing entanglement with motion patterns.
  • Framework: 3Dapter is a multi-view conditioning module trained in a two-stage pipeline to improve fine-grained textures and accelerate convergence.It complements 3DreamBooth by injecting multi-view spatial features through a dual-branch architecture.
  • Evaluation: 3D-CustomBench provides a curated evaluation suite for assessing multi-view-consistent video generation of customized subjects.The benchmark addresses the absence of a standardized dataset for this emerging task.
  • Results: Extensive experiments show that the framework outperforms single-reference baselines and ablation variants in 3D-aware, identity-preserving video generation.The reported comparisons evaluate both 3D consistency and preservation of subject identity.

2 Related Works

Related work extends subject-driven customization from images to videos, but existing approaches trade off speed, detail preservation, and 3D consistency. The proposed framework is positioned as a unified alternative to these paradigms.

  • 3D-Aware Conditioning: 3D-asset and multi-view conditioning remain relatively unexplored for video generation because large-scale pose-aligned object-video datasets are difficult to acquire.Prior work has explored 3D-conditioned images and multi-view text-to-video, but extending these directions remains challenging.
  • Subject-Driven Customization: Subject-driven customization adapts diffusion foundation models to compose user-provided objects into diverse scenes and has expanded from images to videos.Earlier approaches include textual binding, textual inversion, and visual adapters.
  • Customization Paradigms: Training-based zero-shot methods generate rapidly but may sacrifice fine-grained details, whereas optimization-based methods preserve characteristics better but require slow test-time optimization.The paper frames its framework as combining these complementary strengths.

3 Method

3DreamBooth learns a subject’s 3D identity from static multi-view images by restricting training to single frames, while 3Dapter injects visual spatial features to improve detail preservation and convergence. The framework further uses joint attention to select view-specific geometric cues rather than uniformly combining references.

  • 3.2 3DreamBooth: 3DreamBooth treats static multi-view images as single-frame videos, allowing spatial identity learning while bypassing temporal attention and avoiding motion-specific overfitting.A consistent prompt containing the identifier V is used across views, encouraging the identifier and LoRA weights to absorb multi-view geometry.
  • 3.2 3DreamBooth: Multi-view supervision embeds a subject’s geometric variations into the identifier V and the 3DreamBooth LoRA weights.The method samples subject views and optimizes the pre-trained video DiT with trainable LoRA parameters while keeping the original parameters frozen.
  • 3.3 3Dapter: 3Dapter addresses text-conditioning bottlenecks by directly injecting multi-view spatial features into the diffusion process.It is motivated by the slow optimization and loss of high-frequency textures caused by relying primarily on a single text identifier.
  • 3.3 3Dapter: 3Dapter uses single-view pre-training before multi-view conditioning, while a shared adapter processes reference views and joint attention combines their features with the main branch.The shared architecture extracts consistent geometric features without linearly increasing parameter count, and distinct positional indices keep views separate.
  • 3.3 3Dapter: 3Dapter’s visual priors reduce optimization burden, accelerate convergence, and preserve high-fidelity details that text-driven 3DreamBooth struggles to encode.The cited comparison highlights earlier preservation of intricate textures such as “RIO” typography when 3Dapter is integrated.
  • 3.3 3Dapter: Joint attention acts as a dynamic selective router, extracting view-specific geometric hints while filtering conflicting signals from irrelevant references.This mechanism queries only relevant conditioning information for reconstructing the current target view rather than uniformly aggregating all visual features.

4 Experiments

The experiments evaluate 3DreamBooth and 3Dapter against video customization baselines using a curated multi-view benchmark, rotating-video protocols, and fidelity, quality, and alignment measures.

  • 4.1 Evaluation Benchmark: 3D-CustomBench: 3D-CustomBench contains 30 complex objects with high-resolution textures and complete 360° coverage supplemented by custom-captured data.Most objects come from MVImgNet, whose sequences were augmented because they often lacked full orbital coverage.
  • 4.2 Optimization and Conditioning: Each object uses approximately 30 multi-view images for 3DreamBooth optimization and four angularly diverse conditioning views for 3Dapter.Conditioning views maximize angular coverage while minimizing visual overlap.
  • 4.3 Comparisons and Metrics: The framework is compared with VACE and Phantom using quantitative and qualitative evaluations of multi-view subject fidelity.Fidelity is measured with CLIP-I, DINO-I, and GPT-4o-based judging after generating 360-degree rotating videos and isolating foregrounds.
  • 4.3 Comparisons and Metrics: Table 1 compares full and ablation models, with S and M denoting single-view and multi-view conditioning.The comparison targets subject fidelity across views and includes the proposed framework alongside state-of-the-art baselines.
  • 4.3 Comparisons and Metrics: Table 2 measures Chamfer Distance between generated frames and ground-truth multi-view images from 3D-CustomBench.This metric specifically evaluates 3D geometric fidelity.

5 Results

The full 3Dapter+3DB framework preserves subject identity across views and achieves strong geometric fidelity without compromising video quality or text alignment.

  • 5.1 Qualitative Results: The framework synthesizes full 360° geometry and preserves identity across all viewpoints, unlike baselines conditioned only on the first view.Single-view baselines lack spatial information for unseen regions, producing inconsistent textures and geometries during rotation.
  • 5.2 Quantitative Results: The full model performs best across most subject-fidelity metrics, especially GPT-4o evaluations of Shape, Color, Detail, and Overall.VACE has a slightly higher CLIP-I score, while DINO-I and GPT-4o better capture fine-grained multi-view fidelity.
  • 5.2 Quantitative Results: Chamfer Distance is 0.0177 for the proposed framework versus 0.0338 for Phantom, while Completeness reaches 0.0172.The reported Completeness lead indicates stronger recovery of full 360° geometry than single-view methods.
  • 5.2 Quantitative Results: The framework outperforms existing methods in Imaging Quality while maintaining highly competitive ViCLIP scores.These results indicate that stronger 3D conditioning does not compromise the reported intrinsic video quality and text alignment measures.
  • 5.3 Ablation Study: 3Dapter alone provides aesthetic quality but weak 3D consistency, whereas 3DreamBooth alone improves geometry but loses fine-grained texture and CLIP alignment.Joint optimization combines 3DreamBooth’s structure with 3Dapter’s high-frequency feature injection for the best trade-off.

6 Conclusion

The paper concludes that 3DreamBooth and 3Dapter provide an efficient framework for 3D-aware customized video generation with strong geometric fidelity and fast convergence.

  • 6 Conclusion: 3DreamBooth separates spatial identity from temporal motion through 1-frame optimization, embedding subject-specific 3D priors without temporal overfitting.3Dapter complements this by selectively extracting geometric features to preserve intricate textures.
  • 6 Conclusion: Evaluations on 3D-CustomBench report state-of-the-art 3D geometric fidelity and fast convergence for the synergistic approach.The conclusion connects these capabilities to applications in virtual production and advertising.

Supplementary Material: 3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model

Supplementary results evaluate the joint 3Dapter+3DreamBooth framework across diverse object categories and show faithful incorporation of robust 3D spatial priors.

  • Diverse Subject Generation: Additional videos cover a wide variety of object categories beyond the main-paper examples.The supplementary figures present diverse subject-generation results.
  • Diverse Subject Generation: Joint optimization bakes a robust 3D spatial prior into the generation process for each subject.The passage characterizes the resulting supplementary videos as faithful, though the excerpt does not specify individual categories or metrics.

B Ablation Study on 3Dapter Single-View Pre-training

Single-view pre-training is necessary for 3Dapter to provide effective visual conditioning during joint optimization. With pre-training, the module enables accurate, high-fidelity 3D customization within a short optimization period.

  • Pre-training necessity: Without pre-training, randomly initialized 3Dapter lacks fundamental visual conditioning capability and remains ineffective after 400 joint optimization iterations.It produces meaningless noise at initialization and suffers optimization collapse even at 400 iterations.
  • Qualitative behavior: The framework preserves subject identity across diverse object categories, dynamic contexts, and complex scenes with human-object interactions.These qualitative results indicate robustness across varied subjects and prompts.
  • Pre-training necessity: Pre-trained 3Dapter supplies a robust structural foundation for multi-view reference conditioning.The module initially processes multi-view frames under 3D RoPE, producing rotation artifacts before joint optimization refines its behavior.
  • Component complementarity: The full 3Dapter-plus-3DreamBooth framework preserves 3D volume, view consistency, structural precision, and texture fidelity better than either module alone.The comparison includes 3DreamBooth-only optimization at 400 and 1600 iterations.

C.1 Comprehensive Qualitative Ablation

The qualitative ablation shows that 3Dapter and 3DreamBooth provide complementary capabilities. Their combination achieves stronger 3D identity preservation while converging faster than 3DreamBooth alone.

  • Component ablation: 3Dapter alone fails to maintain accurate 3D geometric volume and view consistency because zero-shot conditioning does not build a robust 3D prior.Its limitation is contrasted with the full framework in the qualitative ablation.
  • Component ablation: 3DreamBooth alone preserves multi-view identity after 1600 iterations but lacks structural precision and texture fidelity even with four times more iterations.The baseline captures coarse spatial volume earlier, while accurate identity and view-consistent geometry emerge only after prolonged optimization.
  • Full framework: The full 3Dapter-plus-3DreamBooth configuration achieves accurate identity preservation at 400 iterations through complementary module behavior.The cited ablation compares the full configuration with both individual modules and single-view baselines.
  • Training dynamics: 3DreamBooth-only training requires prolonged optimization to fully converge and refine subject details.The training dynamics are visualized across iterations from 0 through 2000.
  • Full framework: Selective routing accelerates 3D customization convergence and is indispensable for preserving high-fidelity details.The conclusion links the routing mechanism to both optimization speed and detail preservation.
  • Architecture extension: The framework extends to Wan 2.1 DiT-based video generation, preserving intricate 3D identity and complex geometric structures without explicit spatial conditioning.This result is reported across diverse contexts.

D Extensibility of 3DreamBooth to Other DiT Architectures

3DreamBooth’s one-frame optimization paradigm transfers to another DiT-based video model. Applied independently to Wan 2.1, it injects multi-view spatial priors into the subject identifier.

  • Experimental setup: The one-frame optimization paradigm is applied independently to Wan 2.1 without 3Dapter’s explicit spatial conditioning.This isolates the extensibility of the core fine-tuning strategy across DiT architectures.
  • Results: On Wan 2.1, 3DreamBooth binds complex multi-view identity to a unique identifier and reflects those spatial priors in generated videos.The generated sequences preserve intricate 3D identity and complex geometric structures across diverse contexts.

E Discussion and Limitations

The discussion contrasts the modular framework with computationally burdensome alternatives and identifies future extensions. Its modular design is presented as more sustainable and efficient than repurposing task-specific baselines.

  • Limitations and efficiency: Repurposing unified task-specific baselines for multi-view fine-tuning is inefficient, unstable, and can cause out-of-memory failures.These models entangle video generation and visual conditioning, so brute-force joint training risks corrupting generation capabilities.
  • Limitations and efficiency: Baseline training required 128 A100 GPUs for VACE and over 30,000 A100 GPU hours for Phantom, whereas 3Dapter pre-training used four RTX Pro 6000 GPUs for four days.The comparison is presented as evidence for the modular approach’s computational sustainability.
  • Framework behavior: The framework’s ablation combines visual conditioning and optimization to preserve fine-grained identity details more effectively than single-view baselines or individual modules.The convergence analysis also shows that 3DreamBooth requires prolonged optimization to refine subject details.
  • Future directions: Future work includes modeling temporal state changes and extending the optimization framework to video editing with reference videos.These directions target 3D-aware subject insertion in real-world footage.

F Ablation on Visual Conditioning and Convergence

The ablation shows that combining 3Dapter with 3DreamBooth preserves fine-grained identity details more accurately than baselines or isolated modules, while converging faster than text-driven optimization alone. The evaluation protocols measure subject fidelity across views using visual comparisons, point-cloud alignment, feature consistency, and LLM-based judgments.

  • Visual ablation: The combined 3Dapter+3DreamBooth framework achieves more accurate identity and detail preservation than zero-shot baselines or either isolated module.VACE, Phantom, and 3Dapter alone fail to maintain accurate 3D spatial volume and view consistency, while 3DreamBooth alone produces garbled text.
  • Convergence analysis: Text-driven 3DreamBooth optimization requires substantially longer training to refine intricate textures than the full framework.Even at 1600 steps, generated text is less crisp than the full framework’s result at 400 steps.
  • Evaluation setup: The fidelity evaluation generates standardized 360-degree rotation videos to assess multi-view subject preservation.Videos use a prompt that keeps the entire object in frame against a pure white background, isolating subject fidelity from environmental distractors.
  • Geometric fidelity: 3D geometric fidelity is measured by independently lifting generated frames and ground-truth images into point clouds, then aligning them to compute Chamfer Distance.Depth estimation and foreground masking precede a three-stage registration process because the point clouds occupy arbitrary coordinate frames.
  • Feature consistency: Feature-based multi-view consistency averages each generated frame’s maximum cosine similarity to four condition views using CLIP and DINOv2 features.The maximum is selected because an orbiting frame should align strongly with at least one reference view.
  • LLM-as-a-Judge: LLM-based evaluation scores shape, color, detail, and overall identity preservation from 1 to 5 using reference views and sampled generated frames.GPT-4o evaluates four condition views alongside ten uniformly sampled video frames, with temperature set to 0.1.
Loading 2603.18524v1…