Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling

1 Introduction

• Existing methods guided by prompts or single images and videos can fail to preserve a subject’s identity across novel views.L4GM [73], for example, produces side and back views that differ from the reference, including blueish tones on occluded cat regions.Source: §1 · p. 1
• Single-image or video guidance leaves few cues for unobserved views; optimization can be slow and average appearance or motion, while multi-view diffusion can have color and appearance errors.Recent native 3D methods are efficient but still struggle to preserve the reference identity.Source: §1 · pp. 1–2
• TIRE uses long-video tracking to identify regions needing infilling, customized 2D inpainting to match the source subject, and resplatting to return multi-view observations to 3D consistently.The stages work together in a cascade to progressively infill textures.Source: §1 · p. 2
• TIRE improves identity preservation over baseline methods and is presented as complementary to feed-forward and native 3D/4D generation approaches.The paper reports evaluations on its DreamBooth-Dynamic benchmark and in-the-wild data.Source: §1 · pp. 2–3
• The paper frames subject-driven 3D/4D generation as an orthogonal, potentially complementary effort toward personalized assets.Its stated contributions include video tracking, customized inpainting, and reprojection of infilled observations to 3D.Source: §1 · pp. 2–3

3D Generation

• Recent native 3D methods generate representations directly and efficiently, but personalized customization remains largely neglected and identity preservation remains difficult.The passage traces optimization-based SDS, synthetic novel-view reconstruction, multi-view diffusion, and feed-forward generation.Source: §2 · p. 3

4D Generation

• Prior 4D methods progressed from video-SDS optimization to synthetic multi-view videos and later improvements in geometry, consistency, and efficiency.TIRE instead focuses on subject-driven generation to preserve identity in generated assets.Source: §2 · p. 3

Subject-driven Generation

• Subject-driven generation includes personalized 2D methods and prior 3D approaches using personalized models, identity-aware optimization, or subject-specified priors.TIRE differs by using 2D video tracking and inpainting tools to progressively infill occluded regions.Source: §2 · p. 3

3 Method

• Starting from a rough asset and rendered multi-view observations, TIRE tracks regions for infilling, inpaints them, and unprojects the updated views back to 3D.The pipeline is designed for image-guided 3D or video-guided 4D generation.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Preliminary

• The preliminary discussion draws on latent diffusion for image generation and feed-forward reconstruction models that map multi-view images to 3D representations.LGM predicts 3D Gaussians from four views, while L4GM extends feed-forward Gaussian generation to 4D sequences.Source: §3.1 · pp. 4–5

3.2 First Stage: Track

• Backward tracking from target views to the source produces more accurate, better-shaped infilling masks than forward tracking, supporting less grainy inpainting.The method uses CoTracker correspondences and visibility flags to identify source-view points visible in target views.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Second Stage: Inpaint

• A personalized stable-diffusion inpainting model progressively fills masked regions, beginning near the source view and advancing to farther viewpoints.The process uses a ±20° anchor viewpoint before inpainting farther views; after reaching ±90°, that view anchors the remaining viewpoints toward ±180°.Source: §3.3 · p. 6

3.4 Third Stage: Resplat

• Before resplatting, multi-view diffusion refines consistency across inpainted observations while applying mask-aware updates to unseen views.The source view is excluded from refinement, and only the first 30% of the denoising schedule is used; the refined views are then resplatted to Gaussians, a process adaptable to other 3D representations.Source: §3.4 · pp. 6–7
• In the image-to-3D qualitative comparison, TIRE is reported to preserve reference identity better and achieve superior geometry relative to the compared methods.The comparison includes SV3D [90], Wonder3D [54], LGM [86], MeshFormer [50], TRELLIS [104], and Hunyuan3D-v2.5 [38].Source: Figure 4 · p. 7

4.2 Qualitative Evaluation

• Qualitative comparisons show better identity preservation than baselines in image-to-3D and video-to-4D, including against recent TRELLIS and Hunyuan3Dv2.5 image-to-3D methods.The comparisons include rendered views from different viewpoints and timesteps.Source: §4.2 · p. 7
• TIRE also improves geometry over L4GM, with fewer ghosting artifacts attributed to cross-view inconsistency and stages that propagate source pixels and refine multi-view consistency.Resplat uses a mask-aware halfway diffusion-denoising process.Source: §4.2 · p. 8, Figure 6 · p. 8
• Because progressive texture infilling operates on rendered 2D frames, TIRE is presented as applicable across 3D/4D generation methods and representations.The paper also reports extending identity preservation for Hunyuan3D-v2.5 in the appendix.Source: §4.2 · p. 8

4.3 Quantitative Evaluation

• DINO similarity ranks Customize-It-3D highest, although qualitative comparisons find it worse than other methods, while TIRE outperforms L4GM and STAG4D on this metric.The authors caution that DINO similarity is not well suited to the nuances of subject-driven 3D/4D generation.Source: Table 1 · p. 9, §4.3 · p. 9
• Across multiple consistency aspects, TIRE achieves the best subject consistency in VLM-based evaluation, though all methods remain below the perfect score of 4.The evaluation considers shape, color, texture, and facial features.Source: Table 2 · p. 9, §4.3 · p. 9

4.4 User Study

• In a user study with 18 volunteers and 540 scores, TIRE is more subjectively preferred, and Figure 7 reports the highest overall-quality score without disclosing the subject-driven focus.The study used 10 DreamBooth-Dynamic samples and asked participants to consider subject fidelity and cross-view consistency.Source: §4.4 · pp. 9–10, Figure 7 · p. 9

4.5 Ablation Study

• Without progressive infilling, the model can copy source-view appearance into a different pose, such as placing cat-face whiskers on a side view 60° away.The ablation compares against a model fine-tuned only on the source view and its augmentations.Source: §4.5 · p. 10, Figure 8 · p. 10
• A 15% inpainting denoising schedule can leave textures unchanged, while 50% risks overly changing them and reducing realism; the study adopts 30%.The appendix contains additional ablations.Source: §4.5 · p. 10, Figure 9 · p. 10

5 Conclusions

• TIRE combines tracking to find regions for infilling, inpainting to fill occluded regions, and resplatting to return multi-view observations to 3D with cross-view consistency.The conclusion describes the method as complementary to current efficient feed-forward 3D/4D generation research.Source: §5 · p. 10

NeurIPS Paper Checklist

• The authors state that the work’s limitations are discussed in Appendix C and that positive and negative societal impacts are discussed in Appendix G.Source: NeurIPS Paper Checklist · pp. 17, 20
• The authors say the paper and appendix provide details sufficient to understand and reproduce the experimental results.They promise to release the data and code after acceptance, although implementation is absent at submission.Source: NeurIPS Paper Checklist · pp. 18–19
• Training and test details are specified in the paper and appendix, and compute-resource information is provided in Appendix C.Source: NeurIPS Paper Checklist · pp. 19–20
• The paper mainly relies on qualitative comparisons, according to its checklist response on statistical significance.Source: NeurIPS Paper Checklist · p. 19
• The checklist reports adherence to the NeurIPS Code of Ethics and says the crowdsourcing study posed no potential risks to participants.Source: NeurIPS Paper Checklist · pp. 20, 22

A User Study

• Participants rate three methods’ 4D videos from 1 to 10, with a reference image provided and presentation order randomized.Source: A User Study · p. 24
• The scoring criteria cover cross-view consistency, resemblance to the reference subject, and visual quality such as realism and blur.Source: A User Study · p. 24
• The instructions do not reveal the subject-driven-generation objective, which the authors say reduces subjective preference bias toward their method.Source: A User Study · p. 24

B More Details on Datasets

• The dataset contains 23 animable subjects performing diverse activities, with each subject’s model trained from DreamBooth using an SDXL backbone finetuned with LoRA.Source: B More Details on Datasets · p. 24
• Manually created prompts generate subject-specific activity images, which CogVideoX turns into animated videos for evaluation.Source: B More Details on Datasets · p. 24
• The user-study interface presents generated results and a reference front view while asking participants to rate overall quality without naming the study’s target.Source: B More Details on Datasets · p. 25

C Discussions and Limitations

• TIRE offers an option to refine identity in results from feed-forward 3D/4D generators, complementing rather than replacing those models.Source: C Discussions and Limitations · p. 25
• Each compared generator shows viewpoint-specific appearance biases, making exhaustive hand-designed data-filtering rules challenging and laborious.L4GM shows blueish or whitish patterns, SV3D whitish tones and blurry texture, and Hunyuan3D-v2.5 over-smoothed appearance.Source: C Discussions and Limitations · p. 25, Figure D · p. 27
• TIRE converts progressive 3D texture completion into 2D inpainting followed by 3D resplatting, and can operate on implicit representations such as neural radiance fields.Track also propagates pixels along tracking paths to mitigate the difficulty of later inpainting.Source: C Discussions and Limitations · pp. 25–27
• DINO similarity can score methods with incorrect geometry higher when their appearance resembles the source view, motivating evaluation that combines appearance and geometry.The cited failure example is a viewpoint 135° from the source view.Source: C Discussions and Limitations · p. 27
• TIRE takes around 100 minutes on one NVIDIA A100 GPU, which the authors describe as inefficient and inherited from RealFill.Source: C Discussions and Limitations · p. 27

D.1 Training Details

• Denoising is limited to the first 30% of the schedule to preserve novel-view structure; Wonder3D color features guide color correction before resplatting.Progressive inpainting uses 20° viewpoint increments, with 300 training steps per stage.Source: §D.1 · p. 28

D.2 VLM-based Evaluation Details

• VLM prompts follow DreamBench++ concept-preservation evaluation to score identity consistency between a reference image and a rendered generated-asset view.The appendix provides the prompts used for this evaluation.Source: §D.2 · p. 29

### Task Definition

• The evaluator judges semantic consistency between the generated subject and the reference image.Each evaluation receives a reference image and a generated image.Source: ### Task Definition · p. 29

### Scoring Criteria

• Identity scoring considers shape, color, texture, and facial features for people or animals.Texture assessment includes local detail and clarity; facial similarity is especially considered for people and animals.Source: ### Scoring Criteria · p. 29

### Scoring Range

• The evaluator assigns an integer score from 0 to 4, ranging from no resemblance to a virtually indistinguishable subject.The intermediate levels progress from minimal through moderate to strong resemblance.Source: ### Scoring Range · p. 29

### Input Format

• Each evaluation presents two images: the reference first and the generated image second.The evaluator is instructed to review each subject image carefully.Source: ### Input Format · p. 29

Score: [Your Score]

• The required output is only the score, with the evaluator’s analysis excluded.Source: Score: [Your Score] · p. 29

E.1 Degree of Progressiveness for Inpainting

• A 20° progressiveness setting balances tracking-frame count against the difficulty of inpainting regions far from the source view.Larger increments produce worse inpainting in distant regions; smaller increments yield similar visuals but require additional mask-calculation steps.Source: §E.1 · p. 30

E.2 Necessity of the Inpainting Process

• In the inpainting ablation, infilling repairs a large gray patch on a dog’s nose and makes its body texture more realistic and coherent with the source view.Before inpainting, the dog also has messy dark-brown body patterns.Source: §E.2 · p. 31

F More Experimental Results

• The additional experiments cover in-the-wild robustness, application to a recent image-to-3D model, comparisons with state-of-the-art methods, and SV4D under different pose settings.Source: F More Experimental Results · p. 31

F.1 Comparisons on In-the-wild Data

• On in-the-wild L4GM examples, the method more faithfully preserves subject fidelity and achieves superior overall quality compared with the baseline.The examples come from L4GM’s official webpage because its validation dataset is unreleased.Source: §F.1 · p. 31

F.2 General Solution to 3D/4D Generation Methods

• As a plug-in refinement for Hunyuan3D-v2.5, the method improves generated-asset texture so it better matches the reference identity.The design uses 2D tools and can be applied by replacing the initial asset-generation step with another 3D/4D method.Source: §F.2 · p. 31

F.3 Comparisons with More State-of-the-art Methods

• Recent 3D-generation advances still struggle to produce personalized 3D/4D content that preserves subject identity.The comparisons include Hunyuan3D-v1.0, SPAR3D, Sudo AI, Neural4D, MeshFormer, TRELLIS, and Hunyuan3D-v2.5.Source: §F.3 · p. 32

F.4 More Qualitative Comparisons on 3D/4D Generation

• Comparisons with Wonder3D, SV3D, and LGM report better identity preservation and geometry for image-to-3D, with identity-preserving appearance and enhanced geometry in video-to-4D.Source: §F.4 · p. 32

F.5 Comparisons with SV4D

• SV4D’s official code permits rendering only selected views, so the qualitative comparison uses nearby rather than identical viewpoints; the passage reports weaker texture and geometry for SV4D.Source: §F.5 · p. 32

• The authors hope the work encourages personalized 3D/4D asset creation and further research on identity-preserving generation, which they say remains unsolved.Source: G Societal Impact · p. 32
• The authors identify potential digital-forgery risk and possible copyright or ethical issues if the technique is misused.Source: G Societal Impact · p. 32
• Further comparisons report improved fidelity and geometry against advanced methods and Customize-It-3D, STAG4D, L4GM, and SV4D.The SV4D comparison uses close, not identical, viewpoints.Source: G Societal Impact · p. 35, Figure L · p. 37, Figure M · p. 38
• Qualitative comparisons with Hunyuan3D-v2.5, L4GM, SV3D, Wonder3D, and LGM report improved subject identity or fidelity, with geometry gains also reported.Source: G Societal Impact · pp. 33, 36, Figure I · p. 34