PaperlayerView exact revision on arXiv

Source-linked AI summary

PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation

Jialu Li, Mohit Bansal

arXiv:2305.19195v1cs.CVcs.AIcs.CLcs.LG

TL;DR

VLN research has limited photorealistic training environments, constraining the settings available for training and generalization. PanoGen generates text-conditioned panoramic environments and uses them in VLN training; the paper reports state-of-the-art results across R2R, R4R, and CVDN.

  • Problem

    VLN datasets share a limited set of Matterport3D training environments, while collecting diverse photorealistic 3D rooms is difficult and expensive.

    Source: §1 · pp. 1–2

  • Method

    PanoGen captions Matterport3D views, generates text-conditioned images, recursively outpaints coherent panoramas, and uses them for speaker-data pre-training or observation replacement during fine-tuning.

    Source: §1 · p. 2, §3 · p. 4, §4.3 · p. 6

  • Results

    PanoGen achieves new state-of-the-art results on R2R, R4R, and CVDN, including a 2.7% success-rate improvement on the R2R test leaderboard.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    Speaker-data pre-training with PanoGen is especially effective on CVDN, where under-specified dialogue instructions require commonsense knowledge of visual observations.

    Source: §1 · p. 3, §6.1 · p. 8

    The work evaluates PanoGen for VLN and uses Stable Diffusion models trained for inpainting on LAION-Aesthetics v2 5+; further training on room images might improve generation.

    Source: D Limitations and Broader Impacts · p. 14

Abstract

from arXiv · show

Vision-and-Language Navigation (VLN) requires the agent to follow language instructions to navigate through 3D environments. One main challenge in VLN is the limited availability of photorealistic training environments, which makes it hard to generalize to new and unseen environments. To address this problem, we propose PanoGen, a generation method that can potentially create an infinite number of diverse panoramic environments conditioned on text. Specifically, we collect room descriptions by captioning the room images in existing Matterport3D environments, and leverage a state-of-the-art text-to-image diffusion model to generate the new panoramic environments. We use recursive outpainting over the generated images to create consistent 360-degree panorama views. Our new panoramic environments share similar semantic information with the original environments by conditioning on text descriptions, which ensures the co-occurrence of objects in the panorama follows human intuition, and creates enough diversity in room appearance and layout with image outpainting. Lastly, we explore two ways of utilizing PanoGen in VLN pre-training and fine-tuning. We generate instructions for paths in our PanoGen environments with a speaker built on a pre-trained vision-and-language model for VLN pre-training, and augment the visual observation with our panoramic environments during agents' fine-tuning to avoid overfitting to seen environments. Empirically, learning with our PanoGen environments achieves the new state-of-the-art on the Room-to-Room, Room-for-Room, and CVDN datasets. Pre-training with our PanoGen speaker data is especially effective for CVDN, which has under-specified instructions and needs commonsense knowledge. Lastly, we show that the agent can benefit from training with more generated panoramic environments, suggesting promising results for scaling up the PanoGen environments.

1 Introduction

VLN training relies on a small set of photorealistic Matterport3D environments, limiting the diversity of training settings. PanoGen generates text-conditioned panoramic environments and tests two ways to use them, with reported gains across VLN benchmarks.

1 Introduction

  • Matterport3D-based VLN datasets share only 60 training environments, making it difficult to collect diverse photorealistic settings for learning and generalization.Existing environment edits do not add new objects and layouts, while collecting additional captured environments is expensive.Source: §1 · pp. 1–2
  • PanoGen captions Matterport3D views, generates text-conditioned images, and recursively outpaints unseen observations to create coherent panoramic environments.The paper explores using these environments for speaker-data pre-training and observation replacement during fine-tuning.Source: §1 · p. 2, Figure 1 · p. 2
  • Training with PANOGEN environments improves SotA agents on the R2R test leaderboard by 2.7% in success rate and 1.9% in SPL, and increases CVDN test leaderboard goal progress by 1.59 meters, a 28.5% relative gain over previous SotA agents.Source: §1 · p. 3

2 Related Work

Prior VLN work addresses limited training environments through environment-agnostic representations, edits to existing scenes, or additional captured environments. PanoGen instead proposes text-conditioned recursive outpainting for coherent panoramic environments with diverse objects.

Vision-and-Language Navigation

  • VLN agents navigate visual environments from language instructions, and prior work explores sequence models, vision-language transformers, navigation history, graph representations, and proxy tasks.PanoGen builds on the DUET agent.Source: §2 · p. 3

Environment Scarcity in Vision-and-Language Navigation

  • Prior VLN work mitigates environment bias with feature dropout or environment-agnostic representations, edits existing scenes, or adds HM3D environments, but these approaches remain constrained by captured environments.HM3D adds 900 environments, but collecting 3D environments is expensive and limits further scaling.Source: §2 · p. 3
  • PanoGen differs by generating new panoramic environments from text without human annotation, targeting varied appearance while maintaining panorama consistency.The paper contrasts this with prior methods that synthesize views within original environments or edit existing environments.Source: §2 · pp. 3–4

Text-to-Image Generation

  • Diffusion models have been applied to image synthesis, inpainting, and outpainting; prior work also extended them to indoor 3D scene synthesis using geometry and editing prompts.PanoGen instead generates coherent panoramas with diverse objects from detailed text descriptions.Source: §2 · pp. 3–4

3 PANOGEN: Generating Panoramic Environments

PanoGen captions room views from Matterport3D and uses text-conditioned image generation with recursive outpainting to form coherent, diverse panoramas. Its recursive approach supports multiple descriptions across complex indoor scenes.

3 PANOGEN: Generating Panoramic Environments

  • PanoGen addresses panorama complexity and object co-occurrence by captioning Matterport3D views rather than relying on web captions for separate panorama regions.The method discretizes each panorama into 36 views and captions the views separately to capture objects in smaller regions.Source: §3 · p. 4, §3.1 · p. 4

3.1 Room Description Collection

  • BLIP-2 generates detailed captions for the discretized Matterport3D room images, providing text descriptions for subsequent image generation.The paper uses BLIP-2's pretrained cross-modal alignment and zero-shot captioning ability.Source: §3.1 · p. 4

3.2 Generating Panorama with Text-Conditioned Image Outpainting

  • Unlike a prior panorama method conditioned on one full-panorama description, PanoGen uses multiple descriptions to represent complex scenes with diverse objects.The generated set contains one panorama for each panorama in the R2R training environments.Source: §3.2 · pp. 4–5
  • PanoGen recursively outpaints views conditioned on captions and nearby generated images, avoiding discontinuities from stitching independently generated images.Generation starts from a zero-elevation view, rotates to nearby views, and repeats until all 36 panorama views are generated; the resulting set contains 7,644 panoramas stitched from 275,184 images.Source: §3.2 · p. 5

4 Utilizing Panoramic Environments for VLN Training

PanoGen environments enter VLN training through generated instruction-trajectory pairs for pre-training or observation replacement during fine-tuning. The paper describes these methods within a two-stage VLN training setup.

4 Utilizing Panoramic Environments for VLN Training

  • The paper uses generated panoramic environments either to create instruction-trajectory pairs for pre-training or to replace original observations during fine-tuning.Both approaches are intended to mitigate environment scarcity and overfitting to training environments.Source: §4 · p. 5

4.1 Problem Setup

  • At each time step, a VLN agent receives the instruction, current panorama, and navigation history, then selects among navigable locations until it stops or reaches the step limit.Navigable locations include adjacent locations and observed global viewpoints.Source: §4.1 · p. 5

4.2 VLN Training Procedures

  • VLN training has pre-training and fine-tuning stages; pre-training uses MLM, ITM, and single-action prediction, while fine-tuning follows pseudo-interactive demonstrations.During fine-tuning, ground-truth trajectory sampling depends on the agent's current policy.Source: §4.2 · p. 5

4.3 Generating Instructions for Paths in Panoramic Environments

  • Because generated panoramas can differ in layout and appearance from the original environments, the authors train a speaker to generate instructions aligned with the new panoramas.They build the speaker on mPLUG to incorporate cross-modal alignment, and generate 4,675 instruction-trajectory pairs from R2R paths.Source: §4.3 · p. 6

4.4 Learning from Panoramic Environments during Fine-tuning

  • During fine-tuning, the method randomly replaces m% of trajectory observations with PanoGen panoramas to reduce overfitting to limited training environments.Text conditioning preserves similar semantics to the original environments, supporting instruction-observation alignment when the replacement ratio is not large.Source: §4.4 · p. 6

5 Experimental Setup

Experiments evaluate navigation on R2R, CVDN, and R4R, using unseen environments for validation and testing. The setup specifies task metrics and implementation choices for panorama generation, speaker training, and navigation.

5.1 Dataset and Evaluation Metrics

  • Evaluation uses the Room-to-Room (R2R) [2], Cooperative Vision-and-Dialog Navigation (CVDN) [48], and Room-for-Room (R4R) [17] datasets, with 61 training environments, 11 unseen validation environments, and 18 unseen test environments.Source: §5.1 · p. 6
  • The evaluation reports SR, SPL, GP, NE, and TL, using SPL as the main metric for R2R and R4R and GP for CVDN.SR measures stopping within 3 meters of the target; SPL penalizes long paths, while GP measures distance moved toward the target.Source: §5.1 · pp. 6–7

5.2 Implementation Details

  • Panorama generation uses BLIP-2 captions, Stable Diffusion v2.1 for initial views, and Stable Diffusion v1.5 inpainting for rotated views; navigation follows DUET with CLIP-ViT/B-16 features.The speaker is based on mPLUG-base with 350M parameters and a ViT/B-16 visual backbone; the best model is selected on the unseen validation set.Source: §5.2 · p. 7

6 Results and Analysis

PanoGen improves test-leaderboard performance on R2R and CVDN, while qualitative examples and ablations examine panorama coherence, speaker pre-training, replacement during fine-tuning, and the effect of adding more generated environments.

6.1 Test Set Results

  • 2.7% in success rate and 2.9% in SPL on the R2R test leaderboard, plus 1.59 meters in CVDN goal progress (28.5% relative gain over HAMT), accompany new state-of-the-art results on both datasets.The results compare PanoGen with prior state-of-the-art agents.Source: §6.1 · pp. 7–8

6.2 Qualitative Analysis of Panoramic Environment

  • Recursive outpainting produces continuous views that stitch into panoramas, unlike the disjoint views from separately generated discretized images; generated rooms retain varied objects with new appearances and layouts.The examples include a corridor connecting rooms such as a bedroom and bathroom.Source: §6.2 · p. 8, Figure 3 · p. 8

6.3 Effectiveness of Speaker Data in Pre-training

  • Speaker-data pre-training improves R2R success rate by 1.36% and navigation error by 0.2 meters, CVDN validation-unseen goal progress by 0.43 meters, and R4R SPL by 0.58%.The CVDN gain is 7.8% relative; mPLUG speaker data also outperforms EnvDrop speaker data on R2R and R4R success rate and CVDN goal progress.Source: Table 2 · p. 8, §6.3 · p. 9

6.4 Effectiveness of Panorama Replacement in Fine-tuning

  • Randomly replacing trajectory observations with PanoGen panoramas improves R2R success rate by 1.57%, CVDN goal progress by 0.13 meters, and R4R SPL by 2.31%.Replacing 30% of viewpoints performs best; replacing a large ratio can reduce R2R validation-unseen improvements because the trajectory aligns less well with the instruction.Source: Table 3 · p. 8, §6.4 · p. 9, Table 4 · p. 9

6.5 Impact of the Number of Panoramic Environments

  • Training with more PanoGen environments consistently improves performance, and SPL gains remain unsaturated as more generated environments are added.The authors note that each original R2R training panorama can be paired with an additional generated view, doubling the number of panoramas.Source: §6.5 · pp. 9–10, Table 5 · p. 9

7 Conclusion

PanoGen generates text-conditioned panoramic environments through recursive outpainting and explores their use in VLN pre-training and fine-tuning. The paper reports new state of the art on CVDN and R2R.

7 Conclusion

  • PanoGen uses recursive image outpainting to reconstruct missing panoramic observations gradually from text captions, producing coherent panoramas with diverse objects and room layouts.Source: §7 · p. 10
  • The paper investigates two ways to use generated panoramic environments in VLN: pre-training and fine-tuning.Source: §7 · p. 10
  • Learning with PanoGen achieves new state of the art on CVDN and R2R, demonstrating effectiveness for enhancing generalization to unseen environments.Source: §7 · p. 10

Appendix

  • The supplementary material covers datasets, implementation details, additional generated-panorama examples, limitations and broader impacts, and licenses.Source: Appendix · p. 13

A Datasets

The evaluation covers R2R, R4R, and CVDN, which differ in their path construction and instruction formats.

A Datasets

  • The agent is evaluated on R2R, CVDN, and R4R.Source: A Datasets · p. 13
  • R2R provides detailed step-by-step instructions and shortest-path targets, with 61 training environments and 11 and 18 unseen environments for validation and test.Source: A Datasets · p. 13
  • R4R concatenates adjacent R2R paths, making the ground-truth route non-shortest to encourage instruction following rather than taking a direct shortest path.Source: A Datasets · p. 13
  • CVDN uses interactive dialogue with often under-specified instructions, requiring navigation from dialogue history and room commonsense.Source: A Datasets · p. 13

B Implementation Details

The implementation uses diffusion models for panorama generation, an mPLUG-based speaker for instruction generation, and a DUET-based navigation agent.

B Implementation Details

  • Panorama generation captions R2R training views with BLIP-2-FlanT5-xxL, generates initial views with Stable Diffusion v2.1 base, and outpaints rotated views with Stable Diffusion v1.5-outpainting.Generating all environments took two days on six A100 GPUs.Source: B Implementation Details · p. 14
  • The speaker uses mPLUG-base with 350M parameters and a ViT/B-16 visual backbone, trained for four epochs on one A6000 GPU with batch size 16.Speaker training took two days.Source: B Implementation Details · p. 14
  • Navigation uses the DUET architecture with CLIP-ViT/B-16 visual features, pre-training for 150k iterations and fine-tuning for 40k iterations.Both stages use one A6000 GPU and take approximately one day each; the best model is selected on the validation unseen set.Source: B Implementation Details · p. 14

C Qualitative Examples

The qualitative examples contrast PanoGen panoramas with separately generated discretized views and include indoor and outdoor scenes.

C Qualitative Examples

  • Separately generated discretized views produce inconsistent panoramas, whereas PanoGen produces continuous views with reasonable layout and object co-occurrence.Source: C Qualitative Examples · p. 14
  • PanoGen can generate panoramas for both indoor and outdoor environments.Source: C Qualitative Examples · p. 14
  • Outdoor panorama generation may not directly benefit indoor navigation, but demonstrates potential for generating panoramas with other content such as landscapes.Source: C Qualitative Examples · p. 14

D Limitations and Broader Impacts

PanoGen is presented as a way to improve VLN generalization with limited human-annotated training data and reduce retraining when adapting agents to new environments. The work is limited to VLN, while other panoramic embodied tasks remain future directions.

D Limitations and Broader Impacts

  • PanoGen generates panoramic environments for VLN training and is reported to improve agents’ generalization to unseen environments with limited human-annotated data.The authors say it could reduce the effort of retraining agents when adapting to each new environment.Source: D Limitations and Broader Impacts · p. 14
  • The method uses stable diffusion models trained for inpainting on “laion-aesthetics v2 5+”; further training on room images might improve zero-shot generation.The authors identify this as a limitation of their work.Source: D Limitations and Broader Impacts · p. 14
  • The study investigates VLN specifically, while concept learning and grounding in panoramic environments are proposed as possible future applications.The authors say they will explore other tasks in future work.Source: D Limitations and Broader Impacts · p. 14
Loading 2305.19195v1…