SPW-Nav: A Streaming Panoramic World Model for Language-Guided Navigation

• Prior panoramic generators follow predefined trajectories, while interactive world models and vision-language agents mostly use low-level actions and do not interpret scene-content instructions.Source: §1 · p. 1
• Navigating generated panoramas requires resolving instructions from the model's own output, applying rotation without synthesis, bounding poses over long routes, and strengthening translation following with few network evaluations.Source: §1 · p. 1
• SPW-Nav interprets each instruction in the previously generated panorama as displacement or a localized goal, applies rotation by spherical resampling, and expresses translation relative to the previous chunk.Source: §1 · pp. 1–2
• SPW-Nav generates one-minute 2K 360° streams at 26.6 FPS and introduces SPW-NavSet, 233 hours of panoramic video with trajectories and verified instructions.Source: §1 · p. 2
• Noise-tail trajectory learning and opposite-trajectory guidance strengthen translation following, while multi-term memory and a distilled two-step-per-scale autoregressive generator support streaming.Source: §1 · p. 2

• Panoramic image and video methods process 360° scenes, but the cited generators do not follow camera motion along navigation routes.Source: §2 · pp. 2–3
• Camera-controlled panoramic synthesis follows predefined trajectories, while GenDoP generates text-conditioned trajectories but not video.Source: §2 · p. 3
• Interactive world models generate explorable perspective views from latent or keyboard actions, which prescribe camera motion but cannot express scene-content instructions.Source: §2 · p. 3

• Real-world panoramic footage offers diverse scenes but limited camera-motion control and few paired movement instructions, motivating a curated dataset that also uses controllable rendered routes.Source: §3 · p. 3
• The video corpus contains about 18K clips and 233 hours at 1024 × 2048, combining rendered indoor scenes with exact poses and real outdoor street videos with estimated poses.Source: §3 · p. 3
• Movement annotations pair panoramas with explicit trajectory-derived instructions or visible-object goals localized by bounding boxes and verified by vision-language models.Source: §3 · p. 3
• SPW-NavBench contains 100 held-out real-world outdoor clips, each with an initial panorama, ground-truth trajectory, and two instructions.Source: §3 · p. 3

4.1 Preliminaries

• SPW-Nav frames each stream as panoramic segments generated in 33-frame chunks, with each frame an equirectangular 1024 × 2048 panorama.The stream begins from a panorama and extends as new movement instructions arrive.Source: §4.1 · p. 4

4.2 Navigation in the current view

• Instructions are interpreted in the latest generated panorama as either waypoint displacements or localized object goals, then converted into a camera trajectory.A displacement can include up to three waypoints; a goal specifies a bounding box and approach distance.Source: §4.2 · p. 4

4.3 Pose-aligned video generation

• Spherical rotation decoupling resamples panoramas to apply rotation, leaving the video generator to model translation-induced parallax.At inference, the commanded rotation is restored after decoding; rotation is exact up to interpolation.Source: §4.3 · p. 5
• Pose-aligned conditioning expresses chunk poses relative to the preceding chunk’s first frame, keeping translation inputs within the training range.The reference view is recent, and relative translations remain invariant to rigid transformations of the world frame.Source: §4.3 · p. 5
• Noise-tail trajectory learning emphasizes high-noise samples, while opposite-trajectory guidance contrasts commanded translation with its reverse at the coarsest scale.The guidance cancels components shared by both directions and amplifies direction-dependent components; finer scales use the conditional velocity.Source: §4.3 · p. 5

4.4 Streaming generation with memory

• Streaming uses recent and older latent frames plus a fixed first-frame anchor as memory, and denoises chunks coarse-to-fine with a few-step student.After the first chunk, each chunk uses six network evaluations plus two for guidance; training exposes the student to imperfect memory histories.Source: §4.4 · pp. 5–6

5.2 Language-guided navigation

• 56.66° TDE and 351.6 FVD outperform all baselines’ reported minima of 62.07° and 628.8, while ATE favors keyword-matched PanoWorld presets.SPW-Nav also leads on every other measure; its RE is less than one quarter of any baseline’s, and its PSNR loss is under 1 dB versus about 5 dB for both PanoWorld variants.Source: §5.2 · p. 6, Table 1 · p. 7
• Fine-tuning raises held-out explicit-instruction accuracy from 0.663 to 0.925 and goal-instruction accuracy from 0.154 to 0.692; without the panorama, goal localization falls to zero.The backbone interprets all explicit instructions correctly, localizes 62% of visible targets, and matches commanded direction for 98% of motion instructions with 0.03 m mean distance error.Source: §5.2 · p. 7

5.3 Camera-controlled generation

• 13.59° estimated rotation closely matches the 14.01° command; SPW-Nav has the best video quality on all three measures, while PanoWorld-A. has lower ATE and higher MA.Against perspective generators, SPW-Nav has the lowest RE and RPER and the second-best PSNR, LPIPS, and FVD; its comparison with PanoWorld-A. reflects specialization versus generality.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 Applications

• 1.62° loop-closure rotation error beats the next-best 8.91°, and SPW-Nav has the lowest translation closure error at 30.7%.Matrix-Game 3.0 has higher memory gain (0.249 in perspective and 0.204 as a panorama versus 0.173), but closure rotation error exceeds 22° in both settings.Source: §5.4 · p. 8, Table 3 · p. 8
• Instruction switches produce no visible jump, although instruction-following metrics fluctuate across segments and FVD is higher after the first segment.The test spans four instructions over twelve chunks, about 25 seconds; with eight H200 GPUs and sparse attention, generation runs at 26.6 FPS against 16 FPS playback.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 Ablation study

• 43.53° to 24.30° TDE improvement follows adding OTG with NTL; larger guidance weights add little TDE benefit while raising RE and degrading appearance.OTG adds only two network evaluations per chunk, both at the coarsest scale.Source: §5.5 · pp. 8–9
• Without NTL, RE rises from 0.99° to 26.58°, PSNR drops by over 3 dB, and FVD rises from 395.6 to 512.2; OTG then lowers TDE only from 29.15° to 27.85° and further degrades appearance.The ablation indicates NTL is needed for motion accuracy and for the gain from OTG.Source: §5.5 · pp. 8–9
• 31.31° RE and 596.5 FVD in the variant without SRD are worse than the full model’s 1.84° and 308.0, but the comparison also removes NTL.The lower Table 4 row disables both SRD and NTL, so it does not isolate SRD’s effect.Source: §5.5 · p. 9, Table 4 · p. 9

6 Conclusion

• SPW-Nav outperforms existing panoramic generators in camera-following accuracy and video quality, and closes long routes with near-exact rotation.The paper identifies consistency on distant revisits and the trajectory shape of translation as open issues.Source: §6 · p. 9
• One-minute 2K 360° video streams at 26.6 FPS from instructions interpreted in the previously generated panorama.Spherical rotation decoupling applies rotation exactly; pose-aligned conditioning keeps translation inputs bounded, while noise-tail trajectory learning and opposite-trajectory guidance strengthen translation following.Source: §6 · p. 9
• New instructions can be issued during playback because generation runs faster than playback.Source: §6 · p. 9
• Consistency on distant revisits and the trajectory shape of translation remain open; future work proposes pose-indexed 3D memory and more diverse training trajectories.Motion targets could also come from panorama clicks or external planners because the video backbone receives camera trajectories rather than language.Source: §6 · p. 9

A Additional results

• Upward and downward arrows identify metrics where larger and smaller values are better.Source: A Additional results · p. 14

A.1 Zero-shot generalization to unseen scenes

• Figure A1 shows zero-shot streams from unseen training scenes, including indoor halls, city streets, waterfronts, and natural scenes.Each row spans 14 to 30 seconds with evenly spaced frames.Source: §A.1 · p. 14, Figure A1 · p. 14

A.2 One-minute streams

• Figure A2 shows continuous streams of about one minute, each generated from a single panorama.Each row spans 60 to 74 seconds.Source: §A.2 · p. 14, Figure A2 · p. 15

A.3 Navigation accuracy

• Table A1 evaluates the vision-language backbone alone on a held-out test scene and SPW-NavBench navigation instructions.SPW-NavBench contains 40 displacement instructions and 60 visible-target instructions.Source: §A.3 · p. 15, Table A1 · p. 15, §B.1 · p. 16

A.4 Streaming generation

• Table A2 reports streaming metrics over four consecutive instructions in 20 scenes; segment 2 is a pure turn, so its TDE is undefined.Source: §A.4 · p. 15, Table A2 · p. 16

A.5 Comparison with perspective generators

• SPW-Nav has the lowest rotation errors among the compared perspective generators, while its video quality is second only to CameraCtrl.The 60° crop covers about 340 pixels of SPW-Nav’s output, whereas perspective baselines use their full output resolution for one view; HunyuanWorld-Voyager follows translation more accurately through depth-based reprojection.Source: §A.5 · p. 16, Table A3 · p. 16

A.6 Guidance weight

• Table A4 sweeps the OTG weight κ with and without NTL on SPW-NavBench.The supplied passages describe the sweep but do not report its outcomes.Source: §A.6 · p. 16, Table A4 · p. 16

B.1 Protocol

• The long-horizon experiment uses 20 scenes and two closed routes of twelve chunks per scene, each ending at its initial pose.Matrix-Game 3.0 receives each route as keyboard and mouse actions under its official calibration.Source: §B.1 · p. 16

B.2 Metric definitions

• Trajectory direction error (TDE) is the angle between commanded and estimated translation directions; rotation error (RE) measures cumulative rotation error, and RPER is rotation error per interval.Source: §B.2 · p. 17
• Video-generation measures include ΔRot. for the difference between mean estimated and commanded cumulative rotation, ATE for scale-free trajectory shape error, and motion accuracy for intervals following the commanded direction.PSNR on the forward view, LPIPS averaged over four viewing directions, and FVD measure video quality; memory gain compares revisited and non-revisiting views at the same time gap, while closure errors compare final and initial estimated poses.Source: §B.2 · p. 17
• Navigation displacement accuracy requires path error within max(0.15 m, 10%) of the commanded excursion and heading within 5°; goal accuracy requires IoU of at least 0.3 and center bearing within 20°.Source: §B.2 · p. 17

C Limitations and future work

• The multi-term memory of generated frames keeps revisited views consistent and closes long routes with near-exact rotation.Source: C Limitations and future work · p. 17
• Consistency on distant revisits could be made more reliable with explicit memory, such as a pose-indexed 3D scene representation.Advancing memory mechanisms within video generation may also achieve this.Source: C Limitations and future work · p. 17
• More diverse training trajectories could further improve the trajectory shape of translation.Source: C Limitations and future work · p. 17

D Dataset details

• 8,527 indoor videos are rendered in Habitat from 29 configurations across HM3D, Replica, and ReplicaCAD, along paths covering four horizontal translation directions, turns, and composed routes.The configurations comprise three HM3D, ten Replica, and sixteen ReplicaCAD scenes.Source: D Dataset details · p. 17
• 9,609 outdoor MUGEN clips remain after removing static or in-place-rotation clips and making the videos rotation-free; their turning captions are replaced with neutral scene prompts.The distillation stage uses a direction-balanced subset of 2,500 videos.Source: D Dataset details · p. 17
• Annotations cover explicit movements and localized goals, with goal boxes and approach distances verified by two vision-language teachers.The instruction set spans 68 unique goals from four indoor scenes.Source: D Dataset details · p. 17
• Instruction training, validation, and test sets use disjoint scenes; validation and test contain 128 and 132 instructions from scenes with unseen action wording and distances.Two scenes provide training, while the remaining two supply validation and test data.Source: D Dataset details · pp. 17–18
• SPW-NavBench contains 100 clips from 94 MUGEN source videos, with 40 explicit-instruction and 60 goal-instruction clips.Each clip uses an 81-frame window, and the benchmark shares no clips or source videos with video-backbone training or instruction annotations.Source: D Dataset details · pp. 17–18
• 18,136 clips—9,609 outdoor and 8,527 indoor—make up SPW-NavSet, totaling about 233 hours at 30 FPS.Figure D1 shows evenly spaced frames from sample clips, with each row representing one clip and its source labeled.Source: D Dataset details · p. 18, Figure D1 · p. 19

E Implementation details

• The Wan2.2-TI2V-5B backbone is fine-tuned on panoramic video with LoRA, producing 1024 × 2048 videos at 16 FPS; training includes staged autoregressive, pyramid, trajectory-conditioned, and distillation phases.The VAE yields a 64 × 128 latent grid, and decoding pads two columns from the opposite side to preserve continuity across panorama borders; the vision-language backbone is fine-tuned for three epochs on four H200 GPUs.Source: E Implementation details · p. 20

F Baseline details

• Baselines receive different camera and language interfaces: panoramic methods use their specified trajectories or motion presets, while perspective methods receive a 60° crop of the initial panorama.NWM also receives three ground-truth past views; CubeComposer receives no translation or language input and is excluded from the end-to-end language setting.Source: F Baseline details · p. 20

G Streaming generation loop

• Each instruction is interpreted once using the previously generated panorama, then its trajectory is generated chunk by chunk while the model updates memory and applies spherical rotation.After the chunks for an instruction, the current pose and panorama are updated from the final chunk.Source: G Streaming generation loop · p. 20

H Correctness of spherical rotation decoupling

• Spherical rotation decoupling treats rotation-free frames as valid translation-only views and reconstructs the full trajectory by rotating generated frames, up to discrete-resampling interpolation.A pure panoramic rotation is resampling rather than content generation; the resampling neither creates nor removes content.Source: H Correctness of spherical rotation… · p. 21
• Pose-aligned conditioning uses relative poses that remain unchanged under a rigid transformation of the world frame; for rotation-free poses, the relative rotation is identity.All rotation-free poses share the initial orientation.Source: H Correctness of spherical rotation… · p. 21