Can Learning to Answer Spatial Questions Improve Novel-View Generation?

1 INTRODUCTION

• High-level properties such as object identity, attributes, and inter-object relationships complement geometric cues by helping preserve scene content and spatial organization across views.The paper motivates language-based supervision as a way to represent these properties alongside fine-grained geometry.Source: §1 · p. 2
• The paper investigates 3D question answering and related language tasks as a source of semantic and spatial supervision for novel-view generation.The potential of explicitly learning high-level scene properties for generation remains largely unexplored.Source: §1 · p. 2
• Ask4NV adapts a pretrained Mixture-of-Transformers architecture to jointly model 3D language understanding and novel-view generation.It uses VAE latents and complementary ViT features, adds camera geometry to both, and applies modality-aware attention masking for information exchange.Source: §1 · p. 2
• Ask4NV achieves strong novel-view image quality and geometric consistency across multiple settings while performing competitively on 3D language understanding.Ablations also validate its architectural and attention choices.Source: §1 · p. 2

Novel-view generation

• Novel-view generation predicts target images from source images and the camera parameters of both source and target views.The goal is to preserve scene appearance and geometric consistency, including at interpolated and challenging extrapolated viewpoints.Source: §2 · p. 3
• The generation model uses flow matching to predict target-latent velocity from noisy target latents, source latents, and source and target camera parameters.At inference, it initializes target latents with Gaussian noise, integrates the predicted velocity, and decodes the result into images.Source: §2 · p. 3

3D language understanding

• 3D question answering and visual grounding represent language-understanding tasks that reason about scene semantics and spatial relationships.The model answers scene-related questions or identifies and localizes objects referred to by a language description.Source: §2 · p. 3
• For question answering, the model conditions on spatially enriched ViT features and text embeddings to generate an answer autoregressively, with cross-entropy training on answer tokens.The visual grounding task uses a proposal-matching objective whose formulation is provided in the supplementary material.Source: §2 · p. 3

Shared scene properties

• Question-answering supervision addresses scene properties that generation should preserve, and is used during training without requiring question answering at generation time.Examples connect cabinet color, stool position, and chair category to the properties represented in novel views.Source: §2 · p. 3

3 METHODOLOGY

• Ask4NV builds on a pretrained unified multimodal model with specialized generation and understanding transformers, using joint self-attention to connect their representations.The architecture is designed to use 3D language understanding as supervision for novel-view generation.Source: §3 · pp. 3–4, Figure 3 · p. 4

3.1 CAMERA-CONDITIONED NOVEL-VIEW GENERATION

• Camera-aware VAE tokens incorporate Plücker-ray features into source and target latents, while bidirectional attention between clean source and noisy target tokens encourages cross-view correspondence.The ray representation is rearranged to match the latent spatial resolution before being projected with the latents.Source: §3.1 · pp. 4–5
• On ScanNet, layer 26 exhibits the strongest cross-view correspondence across PCK@1, PCK@2, and PCK@4 thresholds.Figure 4 reports these measures across Transformer layers for Lance-FT and Ask4NV.Source: Figure 4 · p. 5

3.2 ENHANCING NOVEL-VIEW GENERATION THROUGH 3D LANGUAGE UNDERSTANDING

• Camera-aware ViT features from the understanding transformer join VAE and text tokens in joint self-attention, with a mask controlling information flow among token groups.Noisy target tokens can attend to source VAE and ViT tokens; source ViT tokens attend to clean VAE tokens but not directly to noisy target tokens.Source: §3.2 · p. 5

3.3 TRAINING OBJECTIVES

• Training combines a flow-matching objective for novel-view generation with 3D question-answering and visual-grounding objectives as auxiliary supervision.The understanding objectives supervise the understanding transformer, whose representations are also used by the generation transformer through joint self-attention.Source: §3.3 · pp. 5–6

• The analysis compares Ask4NV with Lance-FT and Lance-FT + ViT, which are trained only on novel-view generation samples.Lance-FT + ViT includes ViT conditioning and the generation-side attention mask but omits 3D language understanding samples and objectives.Source: §4 · p. 6

Finding 1

• Ask4NV outperforms both generation-only baselines across ScanNet and ScanNet++ at all reported PCK thresholds.The analysis measures correspondence in intermediate VAE representations without direct correspondence supervision during training.Source: §4 · p. 6, Table 1 · p. 6

Finding 2

• Ask4NV achieves higher patch-wise feature similarity than both generation-only baselines across all four datasets.The comparison uses spatially corresponding patch features from generated and ground-truth views in a 3D-aware feature space.Source: §4 · pp. 6–7, Table 2 · p. 6

5.1 EXPERIMENTAL SETTING

• Training supplements ScanNet novel-view data with DL3DV and packs three generation samples for each 3D language understanding sample.Generation samples use 2–4 source views among eight sampled frames; understanding uses 32 uniformly sampled frames per scene.Source: §5.1 · p. 7
• The evaluation uses DL3DV and ScanNet as in-domain datasets and Re10K and ScanNet++ as out-of-domain datasets.The main comparison generates M = 4 target views from N = 4 source views and reports image-fidelity and geometric-consistency metrics.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

• Ask4NV achieves the best PSNR and LPIPS on all four datasets, best ATE across datasets, and consistently strong SSIM, RPEr, RPEt, and MEt3R results.The evaluation includes in-domain DL3DV and ScanNet and out-of-domain ScanNet++ and Re10K.Source: §5.2 · p. 8
• Ask4NV generates sharper novel views while maintaining geometric consistency across diverse camera trajectories.Figure 5 shows qualitative comparisons on in-domain DL3DV and out-of-domain ScanNet++; blue and red cameras mark source and target viewpoints.Source: §5.2 · p. 8, Figure 5 · p. 8
• Ask4NV demonstrates 3D language understanding on question-answering and visual-grounding benchmarks alongside its novel-view generation objective.The evaluation covers SQA3D, ScanQA, ScanRefer, and Multi3DRefer.Source: §5.3 · p. 9, Table 4 · p. 9

5.3 ABLATION AND ANALYSIS

• With N = 2 or N = 3 source views, Ask4NV consistently outperforms Lance-FT on DL3DV.The comparison predicts M = 6 target views for N = 2 and M = 5 for N = 3.Source: §5.3 · p. 9
• Bidirectional attention between clean source and noisy target VAE tokens improves performance on DL3DV.The ablation identifies direct source-target interaction during generation as beneficial.Source: §5.3 · p. 9

• Ask4NV combines ViT conditioning and a tailored attention mask, with joint training improving generation quality and 3D-aware representations.The conclusion identifies improved cross-view correspondence in VAE representations and stronger alignment with ground-truth views in a 3D-aware feature space.Source: §6 · p. 9
• The work uses existing publicly available datasets and identifies potential misuse of synthetic images as a general visual-generation concern.The authors report no additional ethical concerns specific to the proposed methodology.Source: ETHICS STATEMENT · p. 10
• The model formulation and training objectives are in the main paper, with implementation and evaluation details in the supplementary material.The cited supplementary sections cover training configuration, architecture, attention masks, language-understanding implementation, metrics, and baselines.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 DATASET DETAILS

• ScanNet trains and evaluates novel-view generation, with DL3DV added for training diversity and RealEstate10K and ScanNet++ reserved for cross-dataset evaluation.The understanding benchmarks are SQA3D, ScanQA, ScanRefer, and Multi3DRefer, using preprocessed ScanNet-based data.Source: §A.1 · p. 18
• Both tasks express camera poses relative to the first input camera and normalize camera centers by mean pairwise distance, applying the same transformation to 3D points and object boxes.For novel-view generation, the distance uses both source and target camera centers.Source: §A.1 · p. 18

A.2 3D LANGUAGE UNDERSTANDING DETAILS

• For understanding, each ViT token receives a sinusoidal encoding of the average 3D coordinates of its corresponding pixel region.Coordinates are back-projected using depth and camera intrinsics, transformed into a common frame, and the resulting sequence is processed by the understanding transformer; these depth-derived embeddings are not used for generation.Source: §A.2 · p. 18
• The understanding transformer autoregressively answers questions from spatially enriched visual tokens, with cross-entropy applied to the ground-truth answer tokens.The passage specifies the answer-generation objective and input representation.Source: §A.2 · p. 18
• Visual grounding matches a response-position grounding token to object proposals, ranking candidates with a learned no-object option for descriptions without a matching object.Training uses annotated referents as positives; evaluation uses Mask3D proposals and selects the smallest candidate set reaching cumulative probability 0.4.Source: §A.2 · pp. 18–19

A.3 TRAINING AND ARCHITECTURE DETAILS.

• Training uses Adam8bit on 4 A100 GPUs for 728 optimizer steps, with gradients accumulated over 64 micro-steps and a learning rate warmed up to 5 × 10−5 over 40 steps.The learning rate then remains constant; compared methods receive the same optimizer steps and novel-view generation samples.Source: §A.3 · p. 19
• Novel-view generation uses a timestep shift of 4.0 and 0.1 conditioning dropout for text and camera conditions, while the grounding temperature is 0.07 and loss weights are 1.0 for generation and 0.25 for understanding.Source: §A.3 · p. 19
• The pretrained VAE projector is expanded to accept concatenated ray representations, retaining pretrained latent-channel weights and zero-initializing the added ray-channel weights.Inputs are resized to 480 × 480 for the VAE and 476 × 476 for the ViT; temporal padding preserves input-view and latent correspondence.Source: §A.3 · p. 19

A.4 EVALUATION DETAILS.

• Camera-pose evaluation estimates poses from generated target views alone, aligns their centers to ground truth with Sim(3), and reports ATE and relative rotation and translation errors.Multi-view consistency uses rendered DINO feature-map similarity in both projection directions; lower MEt3R scores indicate stronger consistency.Source: §A.4 · pp. 19–20
• The evaluation preserves each baseline’s conditioning requirements while applying a common protocol, and excludes methods using inference-time depth warping or point-cloud rendering.Understanding metrics include exact match and language metrics for question answering and IoU-thresholded accuracy or F1 for grounding.Source: §A.4 · p. 20

B.1 NOVEL VIEW GENERATION

• Prior novel-view methods use camera conditioning, geometric guidance, cross-view consistency, or geometry-aware latents, while this work studies 3D-language supervision and understanding representations for generation.The paper describes joint training on overlapping scene domains and explicit use of the understanding expert’s visual representations during generation.Source: §B.1 · p. 21

B.2 3D LANGUAGE UNDERSTANDING

• Prior 3D understanding work adds geometry through positional embeddings, voxel features, bird’s-eye-view representations, distillation, or feature fusion, while other work combines understanding with reconstruction.Source: §B.2 · p. 21

B.3 UNIFIED MULTIMODAL MODEL

• Unified multimodal models combine understanding and generation through autoregressive prediction, separate visual encoders, diffusion, or Mixture-of-Transformers experts with joint self-attention.Tables 7 and 8 are identified as ablations of joint-training settings for generation and 3D language understanding, respectively.Source: §B.3 · p. 21, Table 7 · p. 22, Table 8 · p. 22

C.1 ABLATION ON JOINT TRAINING DESIGN.

• The ablation varies joint training, generation-side ViT conditioning, and additional understanding-side VAE conditioning; the final configuration uses joint training and Gen-ViT without Und-VAE.The Joint ✗, Gen-ViT ✓ setting corresponds to Lance-FT + ViT and differs from the final model by omitting 3D language understanding training.Source: §C.1 · p. 22
• Ask4NV’s configuration achieves the best image fidelity across all four datasets, while additionally enabling Und-VAE reduces image fidelity on every dataset and worsens most geometric metrics.Table 7 presents the joint-training ablation, with Gen-ViT configurations sharing generation-side camera conditioning.Source: §C.1 · p. 22
• Under joint training, Und-VAE yields modest gains on SQA3D and visual grounding, while Ask4NV performs better on most ScanQA metrics, leaving no consistent understanding-benchmark benefit.Ask4NV maintains broadly comparable understanding performance alongside better novel-view generation quality.Source: §C.1 · p. 22
• Ask4NV reaches Lance-FT’s final cross-view correspondence accuracy substantially earlier and has higher accuracy after the initial training stage across all three thresholds.The analysis measures token-level correspondence using attention-output features at layer 26.Source: §C.1 · p. 23, Figure 6 · p. 23
• Ask4NV’s VAE-token similarity maps show more localized responses around corresponding scene regions than the diffuse responses of Lance-FT.Similarity is computed between a query token’s attention-output feature and tokens in another view, without camera conditioning.Source: §C.1 · p. 23, Figure 7 · p. 23

C.3 ABLATION ON BIDIRECTIONAL ATTENTION BETWEEN VISUAL TOKENS.

• Enabling both visual-token bidirectional attention connections gives the most consistent performance across datasets, though ScanNet favors Noise–Clean attention alone on PSNR, SSIM, and LPIPS.The combined configuration achieves the best results across all reported metrics on DL3DV and RealEstate10K and most metrics on ScanNet++; it is better on ScanNet ATE, translation RPE, and MEt3R.Source: §C.3 · p. 24

C.4 EFFECT OF UNDERSTANDING SUPERVISION.

• QA-only supervision achieves the best PSNR, SSIM, and LPIPS on DL3DV and ScanNet, while grounding-only performs comparably to the combined configuration.The authors suggest that allocating more of the understanding-training budget to question answering benefits generation in this setting; their default retains both tasks to support QA and grounding.Source: §C.4 · p. 24, Table 10 · p. 24

D LIMITATIONS AND FUTURE WORK

• Benefits are demonstrated under evaluated multiview settings; longer camera trajectories and more diverse in-the-wild scenes remain untested directions.The authors identify preserving appearance and geometry across broader viewing conditions as a requirement for this extension.Source: D LIMITATIONS AND FUTURE WORK · p. 25
• Training and evaluation use fixed image resolution, leaving multiple resolutions and higher-resolution outputs outside the demonstrated scope.The authors note that these extensions introduce challenges in preserving fine visual details across views.Source: D LIMITATIONS AND FUTURE WORK · p. 25
• A performance gap remains relative to leading 3D language understanding models, and investigating how supervision benefits scale awaits larger models and more diverse training data.The authors suggest stronger unified backbones may help narrow that gap and provide richer supervision for novel-view generation.Source: D LIMITATIONS AND FUTURE WORK · p. 25