PaperlayerView exact revision on arXiv

Source-linked AI summary

Feedforward Novel View Synthesis for Heterogeneous Cameras

Meng Wei, Cheng Zhang, Boying Li, Yihang Chen, Jianmin Zheng, Hamid Rezatofighi, Jianfei Cai

arXiv:2610.03522v1cs.CV

TL;DR

Feed-forward NVS commonly assumes context and target views share a camera family, although practical multi-sensor systems combine projections and may require unseen target cameras. The paper combines token-level ray geometry with projection-aware positional reasoning; on ScanNet++ it reports state-of-the-art heterogeneous-camera performance and zero-shot generalization to unseen target cameras.

  • Problem

    Most feed-forward NVS methods assume a fixed camera family, while heterogeneous settings require reasoning across projections, potentially including target cameras unseen during training.

    Source: §1 · p. 1

  • Method

    The method combines token-center relative camera encodings with local raymaps for intra-patch ray geometry and projection-aware 2D RoPE in ray-induced angular coordinates.

    Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 6

  • Results

    The method achieves state-of-the-art performance on ScanNet++ under heterogeneous-camera evaluation and improves mixed-camera synthesis with zero-shot generalization to unseen target cameras.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Perspective, fisheye, and panoramic cameras are treated as calibrated samplings of shared ray space rather than separate visual domains.

    Source: §1 · p. 2

    The method relies on calibrated datasets, and large-scale collections with diverse, accurately calibrated camera systems are difficult to collect and curate.

    Source: §5 · p. 9

Abstract

from arXiv · show

Feed-forward novel view synthesis has recently shown promising results from sparse posed images, but most existing methods assume that context and target views share a fixed camera family. This homogeneous-camera assumption breaks in practical multi-sensor systems, where perspective, fisheye, and panoramic cameras may coexist and where the target projection may be unseen during training. We study feed-forward NVS across heterogeneous central cameras and identify a key ambiguity introduced by tokenization: a visual token aggregates a projection-dependent bundle of pixel rays, while existing camera encodings mainly expose absolute rays or token-center relations. To address this, we combine token-center relative Camera Positional Encodings and proposed local raymaps, a token-level representation that explicitly describes the intra-patch ray distribution summarized by each token. We further propose projection-aware 2D RoPE, which replaces raw image-grid coordinates with ray-induced angular coordinates so that relative positional reasoning is aligned across camera projections. Together, these components treat diverse cameras as calibrated samplings of a shared ray space rather than separate visual domains. On ScanNet++ with heterogeneous-camera system, our method improves over camera-conditioned baselines under mixed-camera evaluation and demonstrates zero-shot generalization to panoramic views.

1 Introduction

Most feed-forward NVS methods assume context and target views share a camera family, despite practical systems combining different projections and requiring unseen target cameras. This section motivates reusable local ray geometry and introduces a heterogeneous-camera approach evaluated on ScanNet++.

1 Introduction

  • Most feed-forward NVS methods assume context and target views share a fixed projection family, limiting their fit to systems combining perspective, fisheye, and panoramic cameras.Applications may also require rendering into a geometry different from the input sensors.Source: §1 · p. 1
  • Camera variation is continuous, real multi-camera data include non-geometric factors, and unseen projections call for reusable geometric principles rather than exhaustive memorization.The text identifies exposure, white balance, and calibration noise as non-geometric factors.Source: §1 · pp. 1–2
  • Local ray configurations can recur across camera types, motivating generalization through reusable patch-level geometry rather than discrete camera categories or sampled instances.An unseen full-image projection may contain patch-level ray patterns similar to those observed during training.Source: §1 · p. 2
  • Existing encodings expose absolute pixel rays [5], relative camera geometry [7] [8], or token-center rays [9] [10], but leave each token's intra-patch ray bundle implicit.The method combines token-center relative Camera Positional Encodings [10] with local raymaps and proposes projection-aware 2D RoPE in ray-induced angular coordinates.Source: §1 · p. 2
  • The proposed formulation achieves state-of-the-art performance on ScanNet++ under mixed-camera and zero-shot target-camera evaluation protocols.The protocols are defined for heterogeneous-camera NVS.Source: §1 · p. 2

2 Related Work

Wide-FoV cameras motivate geometry beyond ordinary image coordinates, and feed-forward NVS has expanded from perspective toward panorama synthesis. Yet methods generally assume a fixed camera family, while existing camera encodings leave token-level ray-distribution ambiguity unresolved.

Wide-FoV Cameras Across 3D Tasks

  • Wide-FoV geometry is modeled across reconstruction, 3D prediction, radiance-field rendering, video generation, and world modeling, motivating projection-aware NVS.Examples include panoramic and universal-camera depth methods [13] [14] [15], 3D reconstruction models [16] [17], and panorama or world-generation methods [22] [23] [24] [25] [26] [27] [28].Source: §2 · p. 2

Feed-forward Novel View Synthesis

  • Feed-forward NVS ranges from perspective-focused methods to panorama-specific systems [6] [33], but typically assumes a fixed camera family during training and evaluation.LVSM [5] is given as a recent Transformer-based example for direct target-image synthesis.Source: §2 · p. 3

Camera Positional Encodings

  • Camera encodings include absolute Plücker rays [5], relative camera geometry [7], and ray-based positional encodings [9] [10], but token-center relations do not describe each patch's ray bundle.Local raymaps provide token-level intra-patch ray evidence, complementing cross-token or cross-view relations.Source: §2 · p. 3

3 Method

The method addresses projection-dependent ambiguity in Transformer tokens by exposing both local ray geometry and camera-aware positional relations. It combines local raymaps with relative camera encoding and replaces raw image-grid positions with ray-induced angular coordinates.

3.1 Prerequisites

  • The setting allows context and target projection models to differ, including camera types unseen during training, and assumes central cameras.The backbone concatenates tokenized context observations and target-camera information before self-attention, then decodes target-position tokens.Source: §3.1 · p. 4

3.2 Extending Camera Positional Encodings for Heterogeneous Cameras

  • Heterogeneous-camera tokens can aggregate different ray bundles despite similar center-ray relations, so center-level cues alone leave patch geometry ambiguous.A token’s RGB evidence summarizes an entire patch, while attention-level camera encodings primarily expose its center ray.Source: §3.2 · p. 5
  • Local raymaps expose each token’s intra-patch ray distribution, complementing token-center camera encodings and supporting transfer through reusable local ray patterns.The paper describes cleaner attention locality with local raymaps than with CamRay representations, which attend to learned placeholder tokens rather than geometrically near tokens.Source: §3.2 · p. 6

3.3 Projection-Aware 2D RoPE for Heterogeneous Cameras

  • Projection-aware 2D RoPE replaces raw patch indices with ray-induced angular coordinates, aligning relative positional reasoning across camera projections.Pinhole, fisheye, and panoramic tokens can be mapped to consistent directions on the viewing sphere.Source: §3.3 · p. 7

4 Experiments

The experiments evaluate heterogeneous-camera NVS on ScanNet++ in mixed-camera training-distribution settings and with panoramic targets withheld from training. They compare camera-conditioning strategies, test local raymaps and projection-aware RoPE, and include qualitative evaluations and ablations.

4.1 Experimental Setup

  • ScanNet++ v2 supplies fisheye, undistorted, and panoramic images; panoramas are held out for zero-shot testing, with camera-specific image resolutions.The study uses official training and validation splits; perspective images are 224×336 and panoramas 256×512.Source: §4.1 · p. 7

4.2 Evaluations

  • Evaluation includes fisheye and undistorted homogeneous protocols, randomly mixed context-target camera types, and two panoramic-target protocols using virtual fisheye or DSLR contexts.The panoramic protocols use six virtual-camera contexts or ten real DSLR contexts, respectively; histogram matching is applied to all methods.Source: §4 · p. 7, §4.2 · p. 8
  • The proposed methods achieve the best performance across all three in-distribution protocols, with projection-aware RoPE adding consistent gains, especially for fisheye and mixed-camera evaluation.The comparison uses relative ray-based camera positional encoding as an effective baseline; Table 1 covers the three protocols.Source: §4.2 · p. 8, Table 1 · p. 8
  • Local raymaps transfer better to unseen panoramic geometry than global or center-only ray representations, while CamRay fails in this zero-shot setting.Panoramic views are excluded from training and used only at test time; Table 2 reports the zero-shot comparisons.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Ablations

  • Replacing raw image-index RoPE with projection-aware RoPE improves overall performance when applied to PRoPE, showing applicability beyond the proposed architecture.“Ours (Raw)” omits projection-aware RoPE and has shown gains over other raymap inputs.Source: §4.3 · pp. 8–9, Table 3 · p. 8

5 Conclusion

The method handles multiple projection families in one feed-forward NVS model, improving mixed-camera synthesis and generalizing to unseen target cameras. It relies on calibrated datasets, which are difficult to collect and curate at scale.

5 Conclusion

  • The method improves mixed-camera synthesis and exhibits zero-shot generalization to unseen target cameras.The conclusion attributes this to exposing token-local ray distributions and ray-domain positional coordinates.Source: §5 · p. 9
  • The paper studies a single feed-forward NVS model for different projection families rather than separate camera-specific systems.Source: §5 · p. 9
  • The method relies on calibrated datasets, and collecting and curating large-scale data from diverse, accurately calibrated camera systems is difficult.The authors state that richer heterogeneous-camera data could improve robustness and help scale the setting to multi-sensor applications.Source: §5 · p. 9

A.1 Architecture

All methods use the same lightweight decoder-only LVSM backbone and training schedule, with context views and the target camera represented as patch tokens.

A.1 Architecture

  • The decoder-only LVSM backbone has 6 Transformer layers, 16 attention heads, token dimension 768, patch size 8, and approximately 24.5M parameters.Source: §A.1 · p. 12
  • Context views and the target camera are converted into patch tokens and concatenated into one sequence before self-attention.Source: §A.1 · p. 12
  • All methods train for 80k iterations with batch size 8 on two NVIDIA A100 GPUs, using a cosine learning-rate schedule with 2.5k warmup iterations.Unless otherwise specified, each sample has two context views and one target view; baselines share the backbone, schedule, resolution, and view-sampling protocol.Source: §A.1 · p. 12

A.2 DSLR-to-Panorama sampling and evaluation pair construction

For each ScanNet++ target panorama, the evaluation constructs DSLR context sets that are nearby and cover the panorama’s spherical field of view. It selects views greedily by newly covered solid angle and retains pairs that meet a coverage threshold.

A.2 DSLR-to-Panorama sampling and evaluation pair construction

  • Target panoramas are paired with a fixed-size set of DSLR context images selected to be nearby and to cover as much of the panorama’s spherical field of view as possible.Source: §A.2 · p. 12
  • Candidate DSLR views are filtered by camera-center distance, and their viewing directions are transformed into the target panorama’s local frame.The DSLR forward axis supplies each view’s direction, which is converted to azimuth and elevation.Source: §A.2 · pp. 12–13
  • The panorama is discretized into weighted angular bins, and views are greedily added according to newly covered weighted area.Coverage is assessed for the selected set, which is retained only if it passes the final threshold.Source: §A.2 · p. 13
  • Solid-angle weights prevent uniformly spaced azimuth-elevation bins from over-emphasizing regions near the poles.Because weighted coverage is normalized by total weight, the weights’ absolute scale does not affect the final coverage ratio.Source: §A.2 · p. 13
  • The evaluation uses 5° angular bins, a 2.0 m panorama–DSLR center-distance threshold, K = 10 context images, and coverage above 0.8, yielding 36 valid scenes.Source: §A.2 · p. 13

C Assets License

The appendix identifies the licenses of assets used in the work, including code and data, and points to additional qualitative mixed-camera comparisons.

C Assets License

NeurIPS Paper Checklist

The checklist states that the abstract and introduction’s claims are supported by the method and experiments. It also reports on reproducibility, code availability, statistical reporting, and discussion of limitations.

NeurIPS Paper Checklist

Loading 2610.03522v1…