Source-linked AI summary
Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, Shreyas Kousik, Animesh Garg
TL;DR
Geometric representation learning can benefit from NVS, but feed-forward NVS representations transfer poorly despite the availability of posed-image supervision. SNAP uses a constrained pose-conditioned decoder and latent feature reconstruction. It performs competitively across five tasks, including against geometry-supervised methods, without dense 3D annotations.
Problem
Feed-forward NVS representations transfer poorly to geometric tasks, while large-scale annotated 3D data used by geometry-supervised methods is expensive to collect.
Source: §1 · p. 1Method
SNAP aggregates frozen DINOv3 patch features into pose-free scene tokens, then predicts target-view features with a pose-conditioned local decoder and latent reconstruction objective.
Source: §3 · p. 2, §1 · p. 2, §3.2 · p. 3Results
Takeaways & Limitations
SNAP's results show that restricting decoder capacity and using latent targets can support geometric transfer from NVS without dense 3D annotations.
Source: §1 · p. 2, §6 · p. 9Pretraining relies on posed video sequences and static, textured scenes, and the fixed compute and data budget has not revealed the architecture's full scaling laws.
Source: §6 · p. 9
Abstract
from arXiv · showhide
This paper examines the role of Novel View Synthesis (NVS) in geometric representation learning. In principle, NVS should reason about 3D scene structure, thereby enabling transferable multi-view geometric representations. Yet, existing encoder-based NVS methods yield poor representations. This is not because of a lack of supervisory signal, but rather due to inconspicuous architectural choices: \textit{spatially expressive decoders} that dilute representational capabilities of the scene encoder, and \textit{low-level pixel-space targets} that hinder feature learning. We present SNAP, a self-supervised encoder-decoder transformer that addresses both through a pose-conditioned local decoder and a latent-space reconstruction objective. SNAP is task agnostic, and we show that it is competitive with special-purpose geometry-supervised methods. SNAP also performs competitively against self-supervised representations across five tasks: visual localization, pose estimation, point correspondence, depth estimation, and robot manipulation. Remarkably, SNAP's patch features exhibit emergent viewpoint invariance that approaches heavily supervised models despite lower compute and data budgets. Under camera shifts where standard 2D representations collapse, SNAP degrades more gracefully, revealing that restricting decoder expressivity actively prevents the suppression of transferable geometric structure. https://snap-nvs.github.io
1 Introduction
Geometric representations from annotated 3D data can be rich and transferable, but collecting dense 3D supervision at scale is expensive. NVS offers supervision from posed images, yet its feed-forward representations transfer poorly; SNAP targets decoder expressivity and pixel-space objectives as architectural causes.
1 Introduction
- Dense 3D ground truth is expensive to collect at scale, despite geometry-supervised transformers producing rich, transferable representations.The introduction names DUSt3R [1], MASt3R [2], and VGGT [3] as examples.Source: §1 · p. 1
- NVS can learn from posed images, but feed-forward NVS representations transfer poorly to downstream geometric tasks.Posed images are available across videos, robotics platforms, and structure-from-motion pipelines.Source: §1 · p. 1
- Expressive decoders can absorb spatial reasoning, while pixel-space targets emphasize high-frequency image statistics weakly related to 3D structure or semantics.These choices can leave the encoder's latent representations insufficiently geometrically grounded.Source: §1 · p. 1
- SNAP combines a pose-conditioned local decoder with reconstruction of frozen DINOv3 patch features to address both architectural failure axes.The paper also reports mechanistic analyses of decoder expressivity and prediction space, and evaluation across five tasks against self-supervised and geometry-supervised methods.Source: §1 · p. 2
2 Related Work
Related work spans self-supervised latent prediction, feed-forward NVS, and geometry-supervised models. SNAP combines latent feature targets with a constrained decoder while pursuing geometric transfer without dense 3D supervision.
Self-Supervised Representation Learning
Novel View Synthesis
- Feed-forward NVS includes SRT [16], RUST [17], LVSM [4], RayZer [6], LagerNVS [18], and SVSM [7], but expressive decoders can absorb representational load.SNAP constrains decoder receptive field with a BlockNN mask and uses latent feature targets to prioritize 3D structural awareness.Source: §2 · p. 2
Geometry-Supervised Models
- Geometry-supervised models such as DUSt3R [1], MASt3R [2], VGGT [3], and D4RT [19] rely on large annotated 3D datasets; SNAP approaches their geometric transferability self-supervised.The related-work passage also identifies π3 and Depth Anything 3 [21] as using explicit 3D supervision.Source: §2 · p. 2
3 Method
SNAP combines a pose-free multi-view encoder with a pose-conditioned local decoder to predict target-view features, training on frozen DINOv3 features rather than RGB. Its evaluation freezes the encoder and trains lightweight task-specific heads across five geometric tasks.
3 Method
- SNAP encodes reference views into scene tokens, then uses a pose-conditioned decoder and latent-feature reconstruction objective to predict novel target views.Training uses scenes from RealEstate10K, DL3DV, and Co3Dv2.Source: §3 · p. 2
3.1 Notation
- Reference images, camera-to-world poses, and intrinsics define the multi-view inputs; target poses and intrinsics specify the views to predict.The first camera pose is set as the identity, and φ denotes a frozen DINOv3 featurizer.Source: §3.1 · pp. 2–3
3.2 Pose-Free Multi-View Encoder as a Spatial Backbone
- The pose-free encoder applies full cross-view self-attention to projected DINOv3 patch features, producing scene tokens Z with implicit geometry.RoPE2D encodes each patch’s position within its image.Source: §3.2 · p. 3
3.3 Pose-Conditioned Patch Decoder as a Feature Predictor
- The decoder uses PROPE-conditioned cross-attention to query Z, while a non-overlapping BlockNN mask limits communication among target patches.The selected receptive field is RF = 1, reducing target-query self-attention to a pointwise residual MLP.Source: §3.3 · pp. 3–4
3.4 Latent Reconstruction Objective
- Reconstruction compares normalized predicted and target DINOv3 features using an L1 penalty and spatial finite-difference penalties.The gradient terms preserve spatial fidelity at object boundaries and texture edges, where patch-wise L1 is less discriminative.Source: §3.4 · p. 4
3.5 Multitask Evaluation of Representation
- The encoder is frozen while lightweight task-specific heads evaluate correspondence, visual localization, multi-view pose estimation, depth estimation, and robot manipulation.The manipulation evaluation measures task success under camera shifts from 0.0 to 1.0 radians.Source: §3.5 · p. 4
4 Analysis: Decoder Expressivity and Prediction Target
The analysis identifies decoder expressivity and prediction target as two factors influencing whether multi-view NVS representations retain transferable spatial structure.
4.1 Axis 1: Decoder Expressivity Degrades Geometric Transfer
- Sweeping BlockNN masks from RF 1×1 to 16×16 varies target-view context, and larger decoder receptive fields weaken the encoder’s pressure to capture precise multi-view geometry.At RF 1×1, target-query spatial communication is eliminated; at RF 16×16, the decoder has full target-view self-attention.Source: §4.1 · p. 5
4.2 Axis 2: Prediction Space Determines Representation Quality
- The sweep varies both input and target DINOv3 layers and evaluates feature reconstruction, point correspondence, patch retrieval, and intrinsic dimensionality on RE10K and DL3DV.Source: §4.2 · p. 5
- Higher-layer DINOv3 feature targets improve downstream geometry, although the easiest reconstruction target is the input-layer 9, target-layer 2 pairing.Earlier ViT layers are closer to RGB, while deeper layers provide higher-level structural abstractions robust to lighting and aliasing.Source: §4.2 · p. 5
4.1 Axis 1: Decoder Expressivity Degrades Geometric Transfer
- As decoder receptive field expands, reconstruction error falls while point correspondence, visual localization, and pose estimation performance degrade.The inverse relationship shows that reconstruction quality alone is a poor proxy for representation quality in encoder-decoder NVS models.Source: §4.1 · p. 5, Figure 2 · p. 6
4.2 Axis 2: Prediction Space Determines Representation Quality
- Input layer 4 improves over layer 11 by roughly 5.8% in PCK and 5.2% in MRR, but has ∼53% higher intrinsic dimensionality; SNAP uses layer 11.The choice accepts a small loss in in-distribution accuracy for a more compressed and invariant multi-view representation.Source: §4.2 · p. 6
5 Experiments
SNAP is evaluated with frozen encoder features and lightweight task-specific heads across geometric tasks, including tests on domains unseen by the vision backbone. The experiments examine correspondence, localization, relative pose, viewpoint-shifted manipulation, and relative depth.
5 Experiments
- SNAP uses frozen encoder features with lightweight task-specific decoder heads to test out-of-the-box generalization without task-specific fine-tuning data.Point correspondence and relative pose use pretraining-dataset validation splits, while depth and manipulation test domains unseen by the vision backbone.Source: §5 · pp. 6–7
- Despite full cross-view attention, SNAP uses shorter input sequences than VGGT and reduces key-value costs with grouped-query attention.For 256×256 images and patch size 16, its sequence length is 256N + 8 tokens, versus VGGT’s 1369N + 5; more than 12 SNAP views would be needed to approach the baselines’ attention cost.Source: §5 · p. 7
5.1 Point Correspondence
- SNAP outperforms self-supervised baselines on RealEstate10K correspondence and remains competitive on DL3DV and Hypersim without explicit 3D supervision.On RealEstate10K, it achieves higher sub-patch precision while processing inputs at 256×256 versus VGGT’s 518×518.Source: §5.1 · p. 7, Table 1 · p. 8
5.2 Visual Localization
- SNAP establishes a new frontier among self-supervised models on most visual-localization metrics.The evaluation trains a lightweight regression head on frozen encoder features to predict rotation and translation.Source: §5.2 · p. 8, Table 2 · p. 8
5.3 Relative Pose Estimation
- On RealEstate10K, SNAP achieves high rotation accuracy among NVS methods and is competitive on translation accuracy with self-supervised and NVS methods.The results indicate that its local discriminative structure coexists with global geometric reasoning.Source: §5.3 · p. 8
5.4 Robot Manipulation Under Viewpoint Shift
- Across three MimicGen tasks, SNAP tracks closely with geometry-supervised VGGT and generally outperforms Adapt3R under camera shifts from 0.0 to 1.0 radians.DINOv3 and ResNet collapse rapidly beyond 0.2 radians.Source: §5.4 · p. 8, Figure 5 · p. 9
5.5 Relative Depth Estimation
- SNAP outperforms NVS-based encoders on relative depth, while geometry-supervised VGGT performs better.SNAP is evaluated zero-shot on 7Scenes, a real-world dataset, and demonstrates strong generalization there.Source: §5.5 · p. 9, Table 3
6 Conclusion
SNAP combines a restricted decoder receptive field with latent feature targets to direct spatial reasoning into scene tokens. The paper reports geometric transfer across five tasks without dense 3D annotations, while noting that pretraining and scaling remain bounded by its current data and compute setting.
6 Conclusion
- Current pretraining relies on posed video sequences and static, textured scenes, and the fixed compute and data budget has not revealed the architecture’s full scaling laws.Extending the paradigm to dynamic video is identified as a critical next step.Source: §6 · p. 9
- A BlockNN mask restricts decoder receptive field to bias spatial reasoning into scene tokens, while last-layer DINOv3 targets redirect optimization toward a structured feature space.The paper reports that SNAP rivals geometry-supervised models across five tasks without dense 3D annotations.Source: §6 · p. 9
- The conclusion cautions that representation quality does not always correspond with reconstruction fidelity, emphasizing decoder capacity restriction over reconstruction optimization.The stated pathologies are expressive decoders that absorb spatial reasoning and pixel-space targets that contaminate gradients with low-level statistics.Source: §6 · p. 9
Supplementary Material
Prior work develops feed-forward novel-view synthesis and multi-view geometric representations, while SNAP examines which NVS design choices shape transfer. The related work also distinguishes SNAP from methods that probe 3D awareness or use different training objectives.
A.1 Feed-forward novel-view synthesis
- Feed-forward NVS ranges from sparse-view radiance-field and image-based rendering models to transformers that synthesize views from learned multi-view representations; this work targets downstream transfer, not only view reconstruction.Per-scene optimization methods such as neural radiance fields and 3D Gaussian splatting require a new scene representation at test time.Source: §A.1 · p. 14
A.2 Multi-view representation learning
- Multi-view geometric models predict cameras, depth, pointmaps, tracks, or scene structure, but many strong methods rely on costly explicit 3D supervision; SNAP instead studies self-supervised NVS and its design factors.This complements masked-reconstruction approaches such as Muskie and MuM [14] [48].Source: §A.2 · p. 14
A.3 3D awareness in video and multi-view models
- Prior work probes 3D information in frozen image, video, and multi-view features, whereas SNAP studies how decoder expressivity and prediction targets shape transferable representations.Among concurrent methods, SVSM [7] and LagerNVS [18] use expressive decoders, while D4RT [19] uses a lightweight cross-attention decoder with supervised geometry targets.Source: §A.3 · p. 14
B Implementation Details
The appendix specifies SNAP’s pretraining data and optimization, then details the labels, heads, and evaluation settings used for correspondence and pose tasks.
B.1 Pretraining Setup
- SNAP is pretrained on about 117,000 multiview sequences, using up to six reference views and one target view per sequence, with 16 encoder self-attention layers and eight alternating PROPE cross-attention and masked self-attention blocks.Training uses 40,000 steps, effective batch size 256, AdamW, and EMA weights with decay 0.999 for reported evaluations.Source: §B.1 · p. 15
B.2 Point Correspondence
- Correspondence labels come from frozen RoMa v2 matches on RE10K and DL3DV or depth-and-pose cross-projection on Hypersim; the head maps tokens to a 16 × 16 grid and predicts coordinates from a similarity heatmap.Views use identity camera-to-world poses to isolate the encoder’s intrinsic geometric awareness; the head is trained for 20,000 steps.Source: §B.2 · p. 15
B.3 Visual Localization
- Visual localization regresses a query image’s absolute pose from a reference context as 6D rotation and 3D translation, using alternating self- and cross-attention followed by an MLP.Figure 6 describes evaluation under camera shifts of 0.2 to 1.0 radians, none seen during training.Source: §B.3 · pp. 15–16, Figure 6 · p. 16
B.4 Pose Estimation
- The relative-pose head predicts all input-view poses in one pass using learned view queries, alternating self- and cross-attention, and a regression MLP.It is trained and evaluated on RE10K and DL3DV splits with a Smooth-L1 rotation-and-translation loss.Source: §B.4 · p. 16
C Robot Manipulation
The robotics appendix evaluates visual encoders in a shared policy setup on three MimicGen manipulation tasks, using fixed and wrist-mounted camera images as inputs.
C Robot Manipulation
- The three MimicGen tasks—STACK THREE-D1, SQUARE-D1, and COFFEE-D1—use a Franka Panda arm, 1,000 demonstrations per task, and RGB observations from fixed third-person and wrist cameras.The policy is conditioned only on visual features.Source: C Robot Manipulation · p. 16
- Six encoders are compared: ResNet-18, Adapt3R, DINOv3, Muskie, VGGT, and SNAP; multi-view encoders receive both cameras while single-view encoders process them independently.The encoders are benchmarked with otherwise identical policy and training hyperparameters.Source: C Robot Manipulation · pp. 16–17
- Every encoder conditions the same diffusion-transformer policy, which predicts eight future actions and executes the first four before replanning.Results are reported over 400 rollouts per task and policy.Source: C Robot Manipulation · p. 17
- Training resolution varies by encoder family: 256 × 256 for DINOv3 and SNAP, 224 × 224 for VGGT and Muskie, and 128 × 128 for ResNet and Adapt3R.All encoders use color jitter followed by a random translation crop of ±8 pixels.Source: C Robot Manipulation · p. 17
D.1 Comparison with Frozen Teachers
Comparisons with frozen teachers and V-JEPA test whether SNAP’s geometric transfer depends on its teacher features or on explicit multi-view learning. SNAP improves correspondence across teachers and benchmarks, with depth gains and largely retained but not uniformly improved single-view semantics.
D.1 Comparison with Frozen Teachers
- PCK@0.1 on held-out Hypersim rises from 29.29 to 55.35 with DINOv3, 40.82 to 52.27 with SAM, and 11.01 to 50.67 with MAE.SNAP improves over each teacher across datasets and thresholds; gains shrink at PCK@0.3 for stronger teachers, while MAE improves by 23.35 points there.Source: §D.1 · p. 17
- SNAP outperforms DINOv3 in relative depth on Hypersim and improves depth-estimation accuracy by 15.8% on 7Scenes, with comparable δ1.The table reports AbsRel and δ metrics on Hypersim and 7Scenes.Source: §D.1 · p. 18, Table 5 · p. 18
- SNAP largely retains teacher semantic discriminability, but ADE20K linear-probe results are lower for DINOv3 SNAP than its teacher: 49.87 versus 51.92 mIoU and 84.31 versus 87.27 pixel accuracy.ImageNet k-NN Top-1 changes from 68.45 to 68.81 for DINOv3, 18.06 to 21.46 for SAM, and 30.78 to 28.64 for MAE.Source: §D.1 · p. 18, Table 6 · p. 18, Table 7 · p. 18
D.2 Comparison with V-JEPA
- 55.35 PCK at threshold 0.1 on Hypersim for SNAP exceeds 21.45 for V-JEPA 2.1-L and 29.29 for DINOv3; SNAP leads both baselines across all three datasets and thresholds.The advantage is largest at tight thresholds and narrows at PCK@0.3, where methods approach saturation.Source: §D.2 · p. 19, Table 8 · p. 19
- SNAP’s stronger correspondence performance than V-JEPA indicates an advantage from explicit multi-view constraints beyond feature-space prediction alone.V-JEPA predicts latent features across video observations; SNAP predicts features at arbitrary target viewpoints from a shared scene representation built from posed images.Source: §D.2 · pp. 18–19
E Inference Speed
SNAP offers a memory-efficient option as input views increase, with a trade-off in inference speed against Muskie.
E Inference Speed
- At V = 100, SNAP requires 1.03 GB of peak GPU memory, 40.5% less than Muskie (1.73 GB) and 69.8% less than VGGT (3.41 GB), demonstrating memory efficiency.Source: E Inference Speed · p. 19
- SNAP has 233.38M parameters, approximately 1.3× fewer than Muskie and 3.9× fewer than VGGT.The count includes the DINOv3 backbone.Source: E Inference Speed · p. 19
- At V = 100, SNAP is 2.9× faster than VGGT but roughly 36% slower than Muskie (0.245s vs. 0.180s).The paper attributes Muskie's latency advantage to using fewer multi-view self-attention blocks.Source: E Inference Speed · p. 19
F Qualitative Depth Estimation Results
The qualitative evaluations show SNAP's depth predictions, layer-selection trade-offs, and spatial feature structure across several visual comparisons.
F Qualitative Depth Estimation Results
- With four input images, SNAP produces coherent relative-depth predictions comparable to LagerNVS, while LVSM appears noisier and less consistent.The comparison is not resolution-matched: SNAP is decoded at 256px and LagerNVS at 518px, which likely contributes to LagerNVS's stronger thin-structure predictions.Source: F Qualitative Depth Estimation Results · pp. 19–20, Figure 7 · p. 21
G Landscape Plots
- Mid-layer inputs improve local correspondence but create a performance valley in cycle consistency and variance ratio; deepest-layer inputs provide a robust operating point.The sweep covers mAP/MRR, cycle consistency, and variance ratio; the figure marks superior performance across metrics in yellow.Source: G Landscape Plots · p. 20, Figure 8 · p. 22
H Qualitative Feature Visualizations
- SNAP's feature maps show smooth spatial gradients over surfaces while retaining object boundaries, unlike DINO's flat semantic regions, Muskie's appearance encoding, and LagerNVS's grid artifacts.The maps are high-dimensional features projected into RGB space using PCA.Source: H Qualitative Feature Visualizations · p. 20, Figure 9 · p. 23