PaperlayerView exact revision on arXiv

Source-linked AI summary

3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model

Hyun-kyu Ko, Jihyeon Park, Younghyun Kim, Dongheok Park, Eunbyung Park

arXiv:2603.18524v1cs.CV

TL;DR

Subject customization in video diffusion is challenged by full-sequence training’s computational redundancy and temporal overfitting; 3DreamBooth uses 1-frame optimization with 3Dapter’s multi-view conditioning. On 3D-CustomBench, the framework achieves state-of-the-art 3D geometric fidelity and fast convergence.

  • Problem

    Injecting subject identity into video diffusion is challenging because full-sequence training is computationally redundant and prone to temporal overfitting.

    Source: Isolating Identity from Temporal… · p. 5

  • Method

    3DreamBooth uses 1-frame optimization to learn multi-view subject geometry, while 3Dapter provides multi-view conditioning.

    Source: §3.2 · p. 6, §6 · p. 14

  • Results

    On 3D-CustomBench, the framework achieves state-of-the-art 3D geometric fidelity and fast convergence; its Chamfer Distance is 0.0177 versus Phantom’s 0.0338.

    Source: §5.2 · p. 14, §6 · p. 14

  • Takeaways & Limitations

    The framework’s supported application scope includes virtual production and advertising.

    Source: §6 · p. 14

    Text-driven 3DreamBooth optimization is slow and computationally demanding, and its single-token condition loses fine-grained detail.

    Source: The Bottleneck of Text-Driven… · p. 7

Abstract

from arXiv · show

Creating dynamic, view-consistent videos of customized subjects is highly sought after for a wide range of emerging applications, including immersive VR/AR, virtual production, and next-generation e-commerce. However, despite rapid progress in subject-driven video generation, existing methods predominantly treat subjects as 2D entities, focusing on transferring identity through single-view visual features or textual prompts. Because real-world subjects are inherently 3D, applying these 2D-centric approaches to 3D object customization reveals a fundamental limitation: they lack the comprehensive spatial priors necessary to reconstruct the 3D geometry. Consequently, when synthesizing novel views, they must rely on generating plausible but arbitrary details for unseen regions, rather than preserving the true 3D identity. Achieving genuine 3D-aware customization remains challenging due to the scarcity of multi-view video datasets. While one might attempt to fine-tune models on limited video sequences, this often leads to temporal overfitting. To resolve these issues, we introduce a novel framework for 3D-aware video customization, comprising 3DreamBooth and 3Dapter. 3DreamBooth decouples spatial geometry from temporal motion through a 1-frame optimization paradigm. By restricting updates to spatial representations, it effectively bakes a robust 3D prior into the model without the need for exhaustive video-based training. To enhance fine-grained textures and accelerate convergence, we incorporate 3Dapter, a visual conditioning module. Following single-view pre-training, 3Dapter undergoes multi-view joint optimization with the main generation branch via an asymmetrical conditioning strategy. This design allows the module to act as a dynamic selective router, querying view-specific geometric hints from a minimal reference set. Project page: https://ko-lani.github.io/3DreamBooth/

1 Introduction

The paper addresses 3D-aware customized video generation, where 2D-centric conditioning can fail to preserve subject identity across unseen views. It combines 1-frame 3DreamBooth personalization with multi-view 3Dapter conditioning.

1 Introduction

  • Existing video customization methods often use single-image references [7] or text prompts, leaving subjects rigidly 2D and inconsistent across unseen viewpoints.The paper identifies the lack of explicit multi-view conditioning in video diffusion models as an open frontier for 3D-consistent customization.Source: §1 · p. 3
  • 3DreamBooth fine-tunes the backbone on single frames to learn spatial identity without entangling it with motion, avoiding the need for multi-view video datasets.Single-frame inputs bypass temporal attention pathways and preserve the model’s pretrained motion priors.Source: §1 · p. 3
  • 3Dapter uses single-view pretraining followed by joint multi-view fine-tuning, conditioning target-view reconstruction on a minimal set of reference views.The dual-branch module injects multi-view spatial features while preserving foundational weights, supporting detailed generation and computational efficiency.Source: §1 · pp. 3–4
  • The framework outperforms single-reference baselines and ablation variants in generating 3D-aware, identity-preserving videos; the paper also introduces 3D-CustomBench for evaluation.The introduction reports the comparison without giving quantitative results.Source: §1 · p. 4

2 Related Works

Prior work has extended subject-driven customization from images to videos, while 3D-conditioned image and video generation remains comparatively underexplored. Existing customization approaches trade rapid generation against detail preservation and inference speed.

2.1 3D-Conditioned Image and Video Generation

  • 3D-asset or multi-view-conditioned generation remains largely unexplored: RefAny3D fine-tunes FLUX [21] for images, while MV-S2V [35] trains on large-scale synthetic multi-view videos at significant computational cost.Extending RefAny3D to video is challenging because large-scale pose-aligned object-video pairs are difficult to acquire.Source: §2.1 · p. 4

2.2 Subject-Driven Customization

  • Image customization methods use textual binding, textual inversion [9], and visual adapters [47], and have extended to video diffusion models [19] [40].These extensions enable broader user-controllable functionalities.Source: §2.2 · p. 4
  • Training-based zero-shot methods enable rapid generation but often lose fine detail, whereas optimization-based methods preserve subject characteristics but slow inference through test-time optimization.The paper says its framework unifies both paradigms, accelerating optimization convergence while better preserving 3D identity in synthesized video.Source: §2.2 · p. 4

3 Method

3DreamBooth learns a subject’s 3D identity from single-frame, multi-view images while preserving a video model’s temporal dynamics. 3Dapter adds reference-image conditioning to retain fine detail and route view-specific geometric cues during generation.

3.1 Rethinking DreamBooth for 3D Customization

  • DreamBooth binds a unique identifier to a subject by fine-tuning a model to reconstruct its appearance, but extending this idea to video requires separating spatial representation from temporal dynamics.Source: §3.1 · pp. 4–5

Isolating Identity from Temporal Dynamics.

  • Training on full subject videos is computationally redundant and risks temporal overfitting; with T = 1, temporal attention is bypassed and updates remain spatial.This preserves the model’s pre-trained temporal priors for smooth, view-consistent video generation without explicitly modifying its architecture.Source: Isolating Identity from Temporal… · p. 5

Eliciting the Implicit 3D Prior.

  • Multi-view DreamBooth training uses pre-trained video models’ implicit 3D priors so identifier V absorbs a subject’s geometry and view-dependent appearance across static views.At inference, the enriched token combines with the model’s pre-trained temporal dynamics to generate temporally consistent views.Source: Eliciting the Implicit 3D Prior · p. 6

3.2 3DreamBooth

  • 3DreamBooth pairs each static subject view with the same identifier-and-class prompt, then uses LoRA updates to bake multi-view geometry into the token and network.The pre-trained DiT parameters remain frozen, and the single-frame velocity-prediction objective focuses updates on spatial features.Source: §3.2 · p. 6

The Bottleneck of Text-Driven Customization.

  • Text-driven customization is slow and loses fine-grained detail because a single semantic token bottlenecks high-frequency textures, text, and geometric nuances.The model must map a randomly initialized token to a complex 3D visual manifold without explicit visual hints.Source: The Bottleneck of Text-Driven… · p. 7

3.3 3Dapter

  • 3Dapter is a multi-view conditioning module that injects a subject’s spatial features into generation to preserve identity and intricate details.Source: §3.3 · p. 7

Single-view Pre-training

  • Single-view pre-training teaches 3Dapter from reference-target image pairs to condition contextual target generation on a clean object reference and its prompt.The adapter’s LoRA weights are optimized while the pre-trained model parameters remain frozen; joint attention combines target, reference, and prompt features.Source: Single-view Pre-training · pp. 8–9

Multi-View Conditioning via Joint Optimization

  • With masked conditioning views spanning 360°, joint attention routes view-specific geometric hints to reconstruct target views rather than uniformly combining references.A shared 3Dapter processes the references, while distinct RoPE temporal indices prevent their spatial features from entangling. Its texture priors alleviate optimization burden and support faster convergence and detail preservation.Source: Multi-View Conditioning via Joint… · pp. 9–10

4 Experiments

The experiments introduce 3D-CustomBench and evaluate 3DreamBooth against video-generation baselines using multi-view identity, geometric fidelity, video quality, and text alignment measures. The reported qualitative comparison finds that the full model preserves geometry and fine detail during 360° rotations.

4.1 Evaluation Benchmark: 3D-CustomBench

  • 3D-CustomBench contains 30 objects with complex structures, non-trivial topologies, high-resolution textures, and consistent lighting, supplemented with custom captures for full 360° coverage.For each object, 3DreamBooth uses the full multi-view sequence (approximately 30 images); 3Dapter uses four angularly distributed conditioning views, and GPT-4o generates one challenging validation prompt.Source: §4.1 · pp. 10–11

4.2 Evaluation Setup and Metrics

  • Evaluation measures multi-view identity with CLIP-I, DINO-I, and GPT-4o judgments; geometry with Chamfer Distance accuracy and completeness; and video quality and text alignment with VBench and ViCLIP.The identity judgment rates shape, color, detail, and overall identity on a 1–5 scale; geometric evaluation compares generated and ground-truth-derived point clouds.Source: §4.2 · p. 11

4.3 Implementation Details

  • The framework uses HunyuanVideo-1.5 with LoRA for both modules; 3Dapter is pre-trained for 100K iterations and jointly optimized with 3DreamBooth for 400 iterations.The 3DreamBooth-only ablation is also optimized for 400 iterations; joint optimization uses a learning rate of 1 × 10−4.Source: §4.3 · p. 11
  • In qualitative 360° rotations, 3DreamBooth preserves precise 3D geometry and intricate textures where VACE and Phantom struggle with unseen-view structures and details.Figure 6 illustrates this comparison with examples such as product tags and small accessories.Source: Figure 6 · p. 12

5 Results

The results show that 3DreamBooth with 3Dapter improves multi-view subject and geometric fidelity while maintaining competitive video quality and text alignment. Qualitative comparisons and ablations attribute the gains to combining 3DreamBooth’s 3D structure with 3Dapter’s fine-grained feature injection.

5.1 Qualitative Results

  • Against VACE and Phantom, the framework preserves subject identity across a full 360° rotation, whereas first-view-conditioned baselines produce inconsistent textures and geometry on unseen regions.The comparison uses identical prompts except for the subject identifier versus descriptive text.Source: §5.1 · p. 13

5.2 Quantitative Results

  • The full model achieves the best performance across most multi-view fidelity metrics, especially GPT-4o Shape, Color, Detail, and Overall, while VACE has a slightly higher CLIP-I score.The authors attribute VACE’s CLIP-I advantage to semantic similarity that can overlook incorrect textures and geometric inconsistency in unseen views; they consider DINO-I and GPT-4o more indicative of multi-view fidelity.Source: §5.2 · p. 13
  • 0.0177 Chamfer Distance (CD) nearly halves Phantom’s 0.0338, while Completeness reaches 0.0172, supporting recovery of full 360° geometry.The paper attributes this advantage to multi-view conditioning; single-view methods struggle to infer unseen 3D structure.Source: §5.2 · p. 14
  • The framework maintains comparable or superior video quality and text alignment, outperforming baselines in Imaging Quality while achieving highly competitive ViCLIP scores.Table 3 evaluates intrinsic video quality with VBench and text-video alignment with ViCLIP Score (ViCLIP-L/14).Source: §5.2 · p. 14, Table 3 · p. 14

5.3 Ablation Studies

  • Jointly optimizing 3Dapter and 3DreamBooth yields the best trade-off: 3Dapter alone offers strong aesthetics but lacks 3D consistency, while 3DreamBooth alone improves geometry but struggles with fine textures and CLIP alignment.The paper credits 3DreamBooth with robust 3D structure and 3Dapter with high-frequency feature injection.Source: §5.3 · p. 14

6 Conclusion

3DreamBooth and 3Dapter form an efficient framework for 3D-aware video customization, combining 1-frame spatial optimization with multi-view geometric conditioning. On 3D-CustomBench, the approach achieves state-of-the-art 3D geometric fidelity and fast convergence.

6 Conclusion

  • 3DreamBooth’s 1-frame optimization decouples spatial identity from temporal motion, embedding subject-specific 3D priors without temporal overfitting.The framework is presented as highly efficient.Source: §6 · p. 14
  • 3Dapter acts as a dynamic selective router that extracts relevant geometric features from multi-view conditioning to preserve intricate textures.Source: §6 · p. 14
  • Evaluations on the curated 3D-CustomBench show that the combined approach achieves state-of-the-art 3D geometric fidelity and fast convergence.The authors identify virtual production and advertising as advanced applications.Source: §6 · p. 14

Supplementary Material: 3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model · A Additional Qualitative Results

Additional qualitative examples show that 3Dapter+3DreamBooth preserves customized subjects across diverse object categories and varied video contexts. The framework maintains subject identity under different prompts and challenging interactions while retaining the pretrained video model’s generative capabilities.

A Additional Qualitative Results

B Ablation Study on 3Dapter Single-View Pre-training · C Training Dynamics and Qualitative Ablation

The ablations show that single-view pre-training gives 3Dapter a foundation for successful multi-view optimization, while the qualitative comparisons show that 3Dapter and 3DreamBooth together address weaknesses of either module alone.

B Ablation Study on 3Dapter Single-View Pre-training

  • Without pre-training, 3Dapter must learn image-to-video mapping and complex multi-view alignment from scratch, causing optimization collapse.The randomly initialized module has no fundamental visual-conditioning capability at 0 iterations and fails to converge after 400.Source: B Ablation Study on 3Dapter… · p. 19, Figure A4 · p. 23
  • At initialization, pre-trained 3Dapter processes multi-view references as a sequential signal under 3D RoPE, causing rotation artifacts.The passage describes the positional encoding as temporal and spatial.Source: B Ablation Study on 3Dapter… · pp. 21, 23
  • At 400 joint-optimization iterations, pre-trained 3Dapter enables accurate multi-view alignment and high-fidelity customization, while random initialization collapses.Without pre-training, outputs are meaningless noise at 0 iterations and remain structurally corrupted with identity loss after 400 iterations.Source: B Ablation Study on 3Dapter… · pp. 19, 23, Figure A4 · p. 23
  • Through joint optimization, pre-trained 3Dapter learns to route attention to relevant multi-view cues and extract precise 3D structure.This enables high-fidelity 3D-aware video customization within 400 iterations.Source: B Ablation Study on 3Dapter… · p. 23

C Training Dynamics and Qualitative Ablation

  • 3Dapter alone fails to maintain 3D geometric volume and view consistency compared with the full framework at 400 iterations.The comparison includes 3Dapter alone and the combined 3Dapter+3DreamBooth model.Source: Figure A5 · p. 24
  • 3DreamBooth alone lacks structural precision and texture fidelity even at 1600 iterations, four times the full framework’s 400.The qualitative ablation compares 3DreamBooth alone at 400 and 1600 iterations with the combined model at 400.Source: Figure A5 · p. 24

C.1 Comprehensive Qualitative Ablation · C.2 3DreamBooth Training Dynamics

The qualitative ablation shows that combining 3Dapter with 3DreamBooth preserves identity and detail in fewer iterations than 3DreamBooth alone. Training-dynamics results show that text-driven optimization gradually refines appearance but remains limited in fine-grained detail.

C.1 Comprehensive Qualitative Ablation

  • The full 3Dapter+3DreamBooth framework preserves identity and detail within 400 iterations, while 3DreamBooth alone at 1600 iterations lacks exact structural precision and crisp textures.The comparison attributes the full framework’s result to 3Dapter’s explicit multi-view visual hints.Source: §C.1 · p. 24
  • 3Dapter alone fails to ensure accurate 3D spatial volume and view consistency because zero-shot visual conditioning struggles to build a robust 3D prior.Source: §C.1 · p. 24

C.2 3DreamBooth Training Dynamics

  • The 3DreamBooth-only baseline develops accurate subject identity and view-consistent geometry only with prolonged optimization.Source: Figure A6 · p. 25

C.1 Comprehensive Qualitative Ablation

  • The dynamic selective-routing mechanism accelerates 3D customization convergence and is indispensable for high-fidelity detail preservation.Source: §C.1 · p. 25

C.2 3DreamBooth Training Dynamics

  • At 400 iterations, 3DreamBooth alone captures coarse subject volume but struggles with high-frequency details such as textures, logos, and typography.The training-dynamics frames also show coarse shape and color early in optimization.Source: Figure A6 · p. 25, §C.2 · p. 26
  • By 2000 iterations, text-driven optimization gradually refines appearance, but textures and typography remain poorly reconstructed.This indicates a fundamental limitation of text-driven optimization alone.Source: Figure A6 · p. 25, §C.2 · p. 26

D Extensibility of 3DreamBooth to Other DiT Architectures

3DreamBooth’s one-frame optimization transfers multi-view identity to Wan 2.1 videos without 3Dapter’s explicit spatial conditioning, supporting its extensibility to other DiT backbones.

D Extensibility of 3DreamBooth to Other DiT Architectures

  • The authors independently applied 3DreamBooth’s one-frame training paradigm, without 3Dapter spatial conditioning, to the Wan 2.1 text-to-video model [40].This setup isolates the extensibility of the fine-tuning paradigm.Source: D Extensibility of 3DreamBooth to… · p. 26
  • Without external spatial priors, 3DreamBooth binds multi-view identity to identifier V and reflects it in synthesized videos, demonstrating extensibility to diverse DiT backbones.The generated results are illustrated in Fig. A7.Source: D Extensibility of 3DreamBooth to… · p. 27

E Discussion and Limitations

The discussion argues that decoupled 3Dapter avoids the inefficiency and instability of joint fine-tuning, while one-frame test-time optimization trades convenience for fine-grained 3D fidelity. It also describes semantic interaction transfer and identifies dynamic subjects and video editing as future directions.

Inefficiency of Repurposing Task-Specific Baselines

  • 3Dapter was pre-trained in 4 days on 4 RTX Pro 6000 GPUs, whereas VACE training required 128 A100 GPUs and Phantom reported over 30,000 A100 GPU hours.The paper argues that joint multi-view fine-tuning of task-specific models is inefficient, risks damaging video-generation capabilities, and can cause OOM failures; their unified pathways entangle generation and visual conditioning.Source: E Discussion and Limitations · p. 27

Test-Time Optimization vs. Zero-Shot Generation

  • The method accepts short test-time optimization because aligned multi-view video datasets do not exist and would be difficult to curate, prioritizing fine-grained 3D accuracy over zero-shot convenience.The authors argue that even if such datasets existed, training would require astronomical scale; they position the trade-off as suitable for advertising and commercial content creation.Source: E Discussion and Limitations · p. 27

Semantic Interactions and Future Work

  • 3DreamBooth lets identifier V inherit the semantic interactions of class noun C, but adaptation to articulated subjects or objects with drastic temporal state changes remains open.Future work also includes extending the optimization framework to video editing with reference videos for 3D-aware subject insertion into real-world footage.Source: E Discussion and Limitations · pp. 27–28

E Discussion and Limitations

  • The component ablation shows that only the combined 3Dapter+3DreamBooth preserves fine-grained identity details, while the convergence analysis indicates that optimization must be prolonged to refine subjects.The figure contrasts single-view baselines and individual modules with the combined framework, and shows training loss and qualitative progression across optimization steps.Source: Figure A8 · p. 28

F Ablation on Visual Conditioning and Convergence … H.3 Video Quality and Text Alignment Metrics

The section shows why visual conditioning improves fine-detail preservation and documents the model’s training and evaluation procedures. The protocols assess identity across views, 3D geometry, and video-text alignment.

F Ablation on Visual Conditioning and Convergence

  • In Fig. A8, 3Dapter+3DreamBooth achieves highly accurate identity and detail preservation, while VACE and Phantom struggle with fine-grained details and 3DreamBooth alone at 400 steps captures coarse geometry but produces garbled text.Even at 1600 steps, 3DreamBooth-only generated text remains less crisp than the full framework’s result at 400 steps.Source: F Ablation on Visual Conditioning and… · pp. 28–29

G.1 Base Model and LoRA Configurations

  • The framework uses HunyuanVideo-1.5 [19] with rank-16, alpha-32 LoRA, training approximately 95.62M parameters, or 1.15% of the 8.3B-parameter backbone.Inference generates 720p videos of 81 frames with 50 denoising steps; the compared VACE and Phantom methods use WAN [40] and require retraining the full backbone.Source: §G.1 · p. 29

G.2 3Dapter Pre-training and Dataset Curation

  • Single-view 3Dapter pre-training uses Subjects200K, an identity-consistent dataset with pose, lighting, and background variation, to learn spatial-feature extraction from one reference image.GPT-4o [16] generates subject descriptions and filters misaligned FLUX.1 [21] image pairs; pre-training runs for 100,000 iterations with rank-16, alpha-32 LoRA.Source: §G.2 · pp. 29–30

G.3 Test-Time Joint Optimization and Hardware Setup

  • Per-subject joint optimization removes reference backgrounds and applies rank-16, alpha-32 LoRA to both modules; text-branch adaptation is particularly important for personalization.Optimization takes 400 iterations at learning rate 1 × 10−4 and requires approximately 13 minutes per subject on one NVIDIA RTX PRO 6000 GPU.Source: §G.3 · p. 30

H.1 Multi-View Subject Fidelity and LLM-as-a-Judge

  • For multi-view consistency, generated 360-degree videos use a pure white background; CLIP and DINOv2 [27] compare each frame with its most similar of four condition views, averaging maxima.GPT-4o [16] is prompted with the four condition views and 10 uniformly sampled video frames; each dataset-method pair is evaluated five times independently, with mean and standard deviation reported.Source: §H.1 · pp. 30–31

H.2 3D Geometric Fidelity Alignment Protocol

  • Geometric fidelity is measured by aligning depth- and mask-derived point clouds with RANSAC, ICP, and scale correction, then computing Chamfer Distance from accuracy and completeness.The protocol uses Depth Anything 3 [23] for depth and BiRefNet [54] for foreground masks; qualitative reconstructions show coherent geometry for the framework, versus over-smoothed or fragmented baseline surfaces.Source: §H.2 · pp. 32–33, Figure A10 · p. 33

H.3 Video Quality and Text Alignment Metrics

  • Video-text alignment uses ViCLIP Score, the cosine similarity between a video-level embedding of eight sampled frames and the corresponding validation-prompt embedding.GPT-4o [16] generates one realistic cinematic validation prompt per 3D-CustomBench object, and ViCLIP-L/14 applies temporal attention across the sampled frames.Source: §H.3 · pp. 33–34

Related papers

Loading 2603.18524v1…