Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator

• Training a decoder to reconstruct 3D from multi-view latents requires extensive training and large, hard-to-obtain datasets.The paper argues that ad-hoc decoders also risk falling behind newer 3D foundation models.Source: §1 · p. 2
• Separately trained generators and VAE decoders can be weakly aligned, leaving latents out of the decoder’s input domain and reducing alignment at inference.Single-step rendering-loss approaches do not sufficiently account for the denoising trajectory.Source: §1 · p. 2
• VIST3A stitches a pretrained feedforward 3D reconstruction model to a video VAE by selecting the 3D layer most linearly related to the generator latents.A closed-form linear stitching layer maps the latent space to the retained downstream decoder, with only minor subsequent fine-tuning.Source: §1 · pp. 2–3
• Direct reward finetuning aligns the generator with the stitched decoder using rewards for decoded visual quality and 3D consistency, without labels.The alignment compares video-model outputs with images rendered from generated 3D scenes.Source: §1 · p. 3
• VIST3A consistently outperforms prior text-to-3DGS methods and also supports high-quality text-to-pointmap generation.The stitching scheme applies across multiple video generators and feedforward 3D models.Source: §1 · p. 3

• Score distillation methods optimize each scene, while multi-stage pipelines add engineering overhead and can lack robustness.The related-work discussion covers generation of point clouds, meshes, voxel grids, NeRFs, and 3D Gaussian splats.Source: §2 · p. 3
• Latent diffusion methods finetune 2D generators on multi-view data and decode their latents with a dedicated VAE, rebuilding 3D reasoning and leaving generator-decoder alignment weak.Some object-centric methods trained on curated datasets and controlled camera paths do not handle challenges such as strongly varying scene scale and lighting.Source: §2 · p. 3
• Feedforward reconstruction models such as DUSt3R and MASt3R predict point maps in one pass, while reproducing their 3D capabilities inside VAE training would be difficult and costly.VIST3A reuses models including AnySplat, VGGT, and MVDUSt3R.Source: §2 · p. 3
• VIST3A’s two-stage design first constructs a 3D VAE by stitching a pretrained 3D model as decoder, then aligns the generator through direct reward finetuning.The figure caption describes the alignment stage as encouraging 3D-consistent, decodable latents.Source: Figure 3 · p. 4
• Model stitching joins a network head to another network’s tail through a trainable linear layer; VIST3A applies it to connect a video VAE’s latent space to a foundational 3D model.The passage describes stitching as a way to build hybrid models with minimal degradation and as a practical model-reassembly technique.Source: §2 · p. 4

3.1 MODEL STITCHING FOR 3D VAE CONSTRUCTION

• The method scans 3D-model layers for activations that match encoded video latents, fits a linear stitching map, and retains the downstream layers as the decoder.The selected layer minimizes the mean-squared matching error.Source: §3.1 · pp. 4–5
• The stitched model approximates the original 3D network; self-supervised finetuning uses its predictions as pseudo-targets, and tested model pairings show minimal stitching performance loss.The finetuning adjusts the stitching layer and retained downstream network using weighted ℓ1 losses over the model’s outputs.Source: §3.1 · p. 5

3.2 ALIGNMENT VIA DIRECT REWARD FINETUNING

• Direct reward finetuning aligns the generative model with the stitched decoder, combining a multi-view generative loss with a prompt-only reward that needs no ground-truth images.This performs joint, end-to-end alignment rather than keeping the generator and decoder separate.Source: §3.2 · pp. 5–6
• The reward combines prompt-adherence and visual-quality scores on decoded multi-view images and 3D renders with image-consistency losses between corresponding views.The consistency term uses ℓ1 loss and LPIPS on views rendered from the same camera poses.Source: §3.2 · pp. 5–6
• Reward gradients are backpropagated through denoising, with detached model inputs, randomized timesteps, and sampled gradient-update steps used to stabilize and improve efficiency.The method retains gradients for updates to the next denoising state to stabilize reward propagation at early steps.Source: §3.2 · p. 6

4.1 EXPERIMENTAL SETUPS

• The experiments pair video generators including Wan 2.1 T2V large with feedforward 3D models including VGGT, AnySplat, and MVDUSt3R.Stitched VAEs are finetuned on DL3DV-10K and ScanNet without 3D labels; DL3DV-10K is used for generative-loss alignment.Source: §4.1 · p. 6

4.2 MAIN RESULTS: 3D GENERATION

• Both tested VIST3A variants outperform baselines across all datasets and evaluation metrics on T3Bench, SceneBench, and DPG-Bench.On SceneBench, Imaging Quality exceeds 60 and Coherence exceeds 3.8; on DPG-Bench, scores are mostly above 75 and often approximately 85.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 7
• Participants rank VIST3A first in more than 68% of cases for text alignment and more than 87% for visual quality.Twenty-eight participants ranked five methods on 14 randomly selected samples.Source: §4.2 · p. 7, Table 4 · p. 8
• Qualitative comparisons show VIST3A producing visually compelling, geometrically coherent renderings that follow prompts, and it generates coherent large-scale scenes by extending the number of frames.Figure 4 samples are from T3Bench, SceneBench, and DPG-bench; the long-scene finding is reported without specific training on very long image sequences.Source: §4.2 · pp. 7–8, Figure 4 · p. 8

4.3 MAIN RESULTS: MODEL STITCHING

• Stitching AnySplat onto any tested video model improves RealEstate10K novel-view synthesis over AnySplat alone, while stitched pointmap quality and camera-pose accuracy barely change.The stitched VAEs also outperform earlier VAE-based approaches and surpass Prometheus3D and VideoRFSplat, which use camera poses and large-scale training data.Source: Table 3 · p. 7, §4.3 · p. 9, Table 5 · p. 9

4.4 ABLATIONS

• For Wan+VGGT, lower stitching residuals correspond to better pointmaps, supporting MSE as the layer-selection criterion; CKA is less precise and misses the best-performing layer at index 2.The CKA analysis reports that similarity generally tracks performance degradation as layer index increases, but does not identify the best layer as reliably as MSE.Source: §4.4 · pp. 9–10
• Direct reward finetuning outperforms the pretrained video model and its multi-view-finetuned version, while an integrated approach is more robust to latent-space noise.Each reward component contributes to overall performance; the integrated approach may lead to more consistent 3D reconstruction from generation noise. Additional tests report retained prompt-based camera control and no degradation in VBench video quality.Source: §4.4 · p. 10

5 CONCLUSION

• VIST3A combines video-model generation and feedforward 3D reconstruction through model stitching, then uses reward-based finetuning to produce high-quality, geometrically consistent outputs.The framework also extends generation to pointmaps and depthmaps, without labeled data or massive training runs.Source: §5 · p. 10
• Unlike pipeline alternatives, VIST3A presents model stitching as a way to combine foundational neural networks into end-to-end solutions.Source: §5 · p. 10

A EXTENDED RELATED WORKS

• Pipeline-based methods train generation and reconstruction independently, risking error accumulation and latent-space perturbation sensitivity; per-scene NeRF or 3DGS optimization also raises inference cost.The cited examples include view inconsistency and texture flicker.Source: A EXTENDED RELATED WORKS · p. 19
• Progressive expansion methods use iterative warping and inpainting or video generation, while elaborate multi-stage pipelines add complexity and slow inference.Source: A EXTENDED RELATED WORKS · p. 19
• Direct reward finetuning provides gradient feedback through the full denoising trajectory, making it a natural fit when rewards depend on decoded 3D outputs.Source: A EXTENDED RELATED WORKS · p. 19
• Concurrent VAE work substitutes visual encoders without explicitly measuring representation compatibility and consequently requires extensive retraining; VIST3A stitches at a layer selected for linear transferability.The selected layer yields a well-initialized decoder needing only lightweight finetuning.Source: A EXTENDED RELATED WORKS · p. 19

B.1 MODEL STITCHING

• The stitching layer interpolates video latents to the target dimensions before Conv3D, with model-specific kernels and resolutions and LoRA rank 64, scaling 32.The interpolation restores the video VAE's compressed temporal dimension and adjusts spatial resolution for the 3D model.Source: §B.1 · pp. 19–20
• The stitched VAE uses separately computed output-component ℓ1 losses with weights chosen to balance scales and avoid instability or divergence.MVDUSt3R confidence is weighted 10^-2; VGGT pose is weighted 5 and confidence and gradient regularization 5 × 10^-3; AnySplat confidence is 10^-2, gradient regularization 5 × 10^-3, Gaussian scales 10, and anchor features 0.1.Source: §B.1 · pp. 19–20

B.2 DIRECT REWARD FINETUNING

• The reward combines prompt-based multi-view image quality, 3D representation quality, and consistency between decoded views and their 3D renderings.Quality scores use CLIP and HPSv2.1; consistency combines pixel-level ℓ1 and LPIPS, with a negative sign so maximizing reward minimizes both discrepancies.Source: §B.2 · p. 20
• Reward scores assess both decoded multi-view images and images rendered from the reconstructed 3D representation against the same prompts.Source: §B.2 · p. 20
• To reduce reward computation, the method scores two sampled decoded views and renderings, weighting the two quality terms by 1/16 and consistency by 0.05.Source: §B.2 · p. 20
• Finetuning follows DRTune-style selective gradients: it samples a reduced denoising trajectory, detaches each step's input, and optimizes generative loss minus reward.The sampling range is 10–50 steps, with gradients enabled at K = 2 steps to reduce memory use.Source: §B.2 · p. 21

C.1 TRAINING SETUP

• Layer search uses 200 VGGT, 800 AnySplat, and 3,200 MVDUSt3R DL3DV scenes with 13 views per scene, restricted to encoder layers because deeper-layer MSE values rise.Source: §C.1 · p. 21
• Stitched VAEs train for 50 epochs on DL3DV and ScanNet with 9 or 13 sampled views, while generative models finetune on DL3DV alone.Stitched-VAE batch sizes are 12 for VGGT, 24 for MVDUSt3R, and 12 for AnySplat; the learning rate is 2 × 10^-4. Generative finetuning uses batch size 12, learning rate 1 × 10^-4, and rank-8 LoRA with alpha 16.Source: §C.1 · p. 21

C.2 DETAILED EVALUATION PROTOCOL

• Text-to-3D evaluation uses all 300 T3Bench prompts, 80 SceneBench prompts, and 100 sampled DPG-Bench prompts.Source: §C.2 · p. 22
• For Gaussian-splat evaluation, MVDUSt3R primitives receive 100-step source-view refinement to address observed scale-estimation errors and weak generalization across domains.Assessment renders eight random viewpoints; other compared refinement methods are evaluated after SDS optimization.Source: §C.2 · p. 22
• 3DGS evaluation measures prompt similarity with CLIP, image quality with MUSIQ, aesthetics with a normalized LAION predictor, and alignment, coherence, and style with Unified Reward.Source: §C.2 · p. 22
• Novel-view synthesis uses eight input frames and four target views, padding short sequences by duplicating the last frame and jointly predicting target poses and renderings.Source: §C.2 · p. 22
• Pointmap and camera-pose evaluation uses 13-frame inputs arranged in smooth view-transition sequences and follows Pi3's evaluation code and preprocessing.Source: §C.2 · p. 22

C.3 SELECTION CRITERION FOR 3D MODELS

• The SceneBench ablation compares no finetuning, generative-loss-only finetuning, each reward alone, and both rewards together.Source: Table 6 · p. 23
• Backbone selection prioritizes pretraining-data scale for generalizability: AnySplat represents large-scale Gaussian splatting, MVDUSt3R balances scale and efficiency, and VGGT offers strong performance and scale.Source: §C.3 · p. 23

D.1 IMPACT OF DIRECT REWARD FINETUNING

• Multi-view supervision substantially improves 3D consistency and overall performance over finetuning-free generation, which often yields geometrically inconsistent outputs and visual artifacts.The finetuning-free baseline uses the original pretrained video model; the comparison's multi-view variant was trained for the same wall-clock duration as the reward-based variants.Source: §D.1 · p. 23
• Adding the 3D-consistency reward to multi-view training reduces performance as geometric correctness comes at the expense of detail, whereas the quality reward improves most metrics.The consistency-reward variant produces overly blurred images; the quality reward enhances prompt coherence and aesthetic appeal.Source: §D.1 · p. 23
• The full method achieves the best imaging quality and Unified Reward scores, while aesthetic and CLIP scores remain slightly below Multi-view + Quality.The authors associate the imaging-quality improvement with visually sharp and geometrically faithful 3D representations.Source: §D.1 · p. 23
• The pretrained video model's temporal motion produces severe ghosting, while multi-view-only finetuning suppresses motion but can degrade semantics and image quality.The comparison describes a missing green football and a blurrier car for the multi-view-only baseline; direct reward tuning yields sharper, more prompt-faithful renderings.Source: §D.1 · pp. 24–25, Figure 7 · p. 24

D.2 BENEFITS OF INTEGRATED VS. SEQUENTIAL 3D GENERATION

• The stitched VGGT consistently outperforms sequential decode-and-reconstruct on ETH3D under latent noise, including at α = 1e−4 to 2e−4.At those noise levels, visual artifacts are hardly perceptible; the authors report that the sequential path's VAE decoder amplifies errors.Source: §D.2 · p. 25

E ADDITIONAL RESULTS

• VIST3A's Wan 2.1 backbone translates camera prompts into a high-angle view or a left-to-right viewpoint sweep in 3D generation.The examples use “Aerial dronshot” and “Camera pans left to right” prompts.Source: E ADDITIONAL RESULTS · pp. 25–26, Figure 9 · p. 26
• The VIST3A-adapted model scores higher than the original video generator on VBench quality and total score, with only a marginal Semantic Score difference.The reported overall VBench performance is slightly improved under the VIST3A setup.Source: E ADDITIONAL RESULTS · p. 26, Table 7 · p. 26
• Early 3D-model layers have lower latent-transfer MSE, but absolute MSE does not predict performance across VAE architectures, so stitching layers must be selected for each model pair.Figure 10 reports the log-MSE comparison; the text contrasts CogVideoX + AnySplat's 0.008 MSE and 21.32 PSNR with SVD + AnySplat's 0.012 MSE and 21.48 PSNR.Source: E ADDITIONAL RESULTS · pp. 26–27, Figure 10 · p. 27

F LIMITATIONS

• The video-derived encoder is not guaranteed to handle arbitrarily unordered multi-view images; its inputs must form a coherent sequence simulating smooth view transitions.The limitation follows from the encoder being inherited from a video-generation model designed for sequential, temporally coherent input.Source: F LIMITATIONS · p. 27