Source-linked AI summary
Bridging Language and Spherical Space: Object-Centric Control for Text-to-Panorama Generation
Derui Li, Qian Qiao, Yuhao Sun, Wenhao Guo, Peng Lu
TL;DR
Existing text-to-panorama methods struggle to ground object-level directional language in spherical scenes, while manual spatial conditions reduce language-based flexibility. PanoCtrl parses text into object-level spherical conditions and injects them into diffusion, achieving state-of-the-art spatial alignment and image quality on the proposed benchmark. Its attention-based conditioning remains insufficient for precise ERP positional control without the complementary spatial branch.
Problem
Existing text-to-panorama methods rely on implicit spatial reasoning and struggle to align object-level directional descriptions with ERP-based spherical representations.
Method
PanoCtrl uses PanoParse to predict object semantics and spherical BFoV parameters, then PanoControl injects semantic and spatial guidance into a diffusion transformer.
Results
PanoCtrl achieves state-of-the-art spatial alignment and image quality, including OPR 98.59, RTA 36.91, and SLE 25.34 on PanoGround.
Takeaways & Limitations
Explicit language-to-spherical modeling supports controllable panoramic generation with object-level directional grounding and high visual quality.
Takeaways & Limitations
Attention-based conditioning operates implicitly through global token interaction and is insufficient for precise positional control in ERP space without spatial residual enhancement.
Abstract
from arXiv · showhide
Panoramic image generation is increasingly important for immersive applications such as virtual reality, augmented reality, and 3D content creation. Unlike perspective images, panoramic images represent a viewer-centered $360^\circ$ surrounding space, where directional expressions such as left, right, front, and behind play a central role in spatial understanding. However, existing text-to-panorama methods largely rely on implicit spatial reasoning and often fail to faithfully ground object-level directional descriptions in spherical panoramic scenes. A straightforward alternative is to introduce explicit layouts, but requiring manually specified spatial conditions reduces the flexibility of language-based interaction and does not directly resolve the misalignment between egocentric directional language and panoramic image space. To address this issue, we propose PanoCtrl, an object-centric framework for controllable text-to-panorama generation. Our method explicitly bridges natural language and spherical panoramic space by converting textual descriptions into structured object-level spherical conditions and integrating them into the diffusion process. Specifically, we introduce PanoParse, a text-conditioned parser that predicts object semantics and spherical bounding field-of-view (BFoV) parameters, and \textbf{PanoControl}, which injects object-level semantic and spatial guidance into the diffusion transformer through object-aware attention and spatial residual enhancement. To support this task, we construct PanoGround, a dataset with object-level spherical annotations and diverse directional descriptions for controllable panoramic generation. Extensive experiments demonstrate that PanoCtrl achieves state-of-the-art performance in both spatial alignment and image quality.
1 Introduction
Panoramic generation requires grounding viewer-centered directional language in spherical space, but existing methods struggle with object placement and ERP alignment. PanoCtrl addresses this with structured object-level conditions, dedicated control modules, and the PanoGround benchmark.
- Panoramas encode a full 360° viewer-centered environment, making directions such as left, right, front, and behind fundamental spatial semantics.
- Existing text-to-panorama methods often generate plausible images while misplacing objects or producing spatial inconsistencies with directional descriptions.
- Explicit layouts are difficult to transfer from perspective images because panoramas involve ERP distortion, spherical geometry, and wrap-around continuity, while manual layouts reduce language flexibility.
- PanoCtrl converts text into object-level spherical conditions and injects semantic and spatial guidance into the diffusion transformer.
- PanoGround provides object-level spherical annotations and a benchmark for controllable panoramic generation.
2 Related Work
Related work spans multi-view stitching and panoramic-aware ERP modeling, but object-level directional grounding remains limited. Existing controllable-generation methods generally assume perspective Euclidean coordinates or require predefined spatial conditions, motivating text-derived spherical control.
- Panoramic generation methods either synthesize and stitch multiple perspective views or model panoramas directly in the equirectangular domain.
- Multi-view approaches can suffer object duplication, structural discontinuities, and weakened global coherence across views.
- Most text-to-panorama methods rely on global text conditioning and implicit spatial reasoning, limiting object-level directional grounding.
- Existing controllable-generation methods are mainly designed for perspective images with Euclidean coordinates and cannot directly handle panoramic spherical geometry and ERP distortion.
- PanoCtrl predicts object-level spherical conditions from text and integrates them into diffusion for explicit directional control.
3.1 Overview
PanoCtrl converts directional language into structured spherical object representations, then injects unified object controls into diffusion through semantic and positional branches. This design targets accurate directional grounding while maintaining visual quality.
- The framework consists of two modules: PanoParse and PanoControl.
- PanoParse predicts each object’s semantic category and spherical BFoV, converting implicit directional descriptions into explicit spatial representations.
- PanoControl encodes object semantics and spherical geometry into unified control tokens for diffusion-transformer conditioning.
- The design is reported to enable accurate object-level directional grounding and end-to-end spatial controllability while maintaining high visual quality.
3.2 PanoParse
PanoParse formulates text-to-panorama understanding as spherical set prediction. It decodes variable numbers of objects from text, predicts semantic and BFoV attributes, and trains with matched classification and localization supervision.
- PanoParse treats text-to-panorama understanding as set prediction in spherical space rather than relying on implicit generative reasoning.
- Learnable object queries attend to text features and capture potential object instances, with a no-object class supporting varying object counts without predefined ordering.
- Each decoded object predicts a semantic category and spherical BFoV attributes comprising center longitude, center latitude, and horizontal and vertical angular spans.
- BFoV centers use coarse-to-fine angular-bin classification followed by continuous offset regression, while angular spans are directly regressed.
- DETR-style bipartite matching applies classification and localization losses to matched queries, while unmatched queries receive no-object supervision.
- The parsing loss supervises object semantics and spherical locations, providing structural guidance for language-to-spherical parsing.
3.3 PanoControl
PanoControl injects parsed object semantics and spherical geometry into the diffusion transformer through complementary attention and region-level residual guidance. This design addresses the limits of implicit global conditioning for precise ERP-space placement.
- Object Condition Construction: PanoControl uses object tokens as a unified interface between parsed semantics and spherical geometry for diffusion control.Only valid object tokens are retained, with eight object queries used by default as a trade-off between representation capacity and generation stability.
- Object-aware Attention Injection: Object-aware attention injects gated object tokens into text and image streams, improving consistency between generated object appearance and textual descriptions.The attention branch conditions both streams but operates mainly through global token interaction.
- Object-aware Attention Injection: Attention-based conditioning is insufficient for precise ERP-space positioning because its guidance remains implicit through global token interactions.This limitation motivates a separate spatial residual enhancement branch.
- Spatial Residual Enhancement: Spatial residual enhancement samples BFoV-aligned local features, applies local feature interaction, and aggregates enhanced features back onto the global ERP token grid.Differentiable BFoV parameters define spherical regions, while overlapping object contributions are accumulated during aggregation.
- Spatial Residual Enhancement: Compared with global token interaction, BFoV sampling and aggregation provide more direct region-level guidance for reducing mismatch between egocentric descriptions and ERP-space generation.The conditioned text tokens and enhanced image tokens are then passed to later diffusion-transformer blocks.
3.4 Training Objectives
PanoCtrl jointly trains spherical parsing and diffusion generation, combining global denoising with object-region supervision. The object-region term focuses denoising capacity on annotated BFoV regions to improve local fidelity and spatial consistency.
- Joint Training: PanoCtrl is trained end-to-end by jointly optimizing the parsing objective and diffusion objective.PanoParse receives structural supervision for object semantics and spherical locations from object-level annotations.
- Diffusion Objective: The diffusion objective combines global denoising loss with an object-region loss weighted by λ_obj.The conditioning includes the prompt and object-level control signals.
- Diffusion Objective: Ground-truth BFoV masks define regions for an additional masked loss that strengthens local object consistency.M_gt denotes the union of ground-truth BFoV regions.
- Joint Training: The object-region loss encourages greater denoising capacity in BFoV-specified regions, improving local object fidelity and spatial consistency.The full model jointly optimizes parsing and diffusion, with λ balancing structured parsing and controllable image generation.
3.5 PanoGround Dataset
PanoGround is a panoramic dataset designed for object-level spherical supervision in controllable generation. It combines large-scale ERP imagery, spherical BFoV annotations, and varied directional descriptions.
- Dataset Construction: PanoGround contains 12,688 ERP panoramic images and 37,980 annotated records spanning 106 object categories.The dataset is assembled from LayerPano3D-PanoData, Matterport3D, and PANDORA.
- Dataset Construction: Each record includes a panoramic image, scene description, object set, and spherical BFoV annotations.Annotations are produced through vision-language scene understanding, open-vocabulary detection, language-model caption refinement, and human verification.
- Dataset Purpose: PanoGround provides object-level spherical annotations and diversified directional descriptions for text-to-layout learning and controllable panoramic generation.These properties distinguish it from existing panoramic datasets within the paper’s stated scope.
4 Experiments
Experiments evaluate PanoCtrl on PanoGround using spatial-alignment and image-quality metrics against recent text-to-panorama baselines. The method achieves strong quantitative alignment and image-quality results, with qualitative comparisons showing more accurate object placement and coherent layouts.
- Dataset and Evaluation: PanoGround contains 12,688 panoramic images and 37,980 object-level spherical BFoV annotations, split into 10,820 training and 1,868 testing panoramas.The corresponding annotated-record split is 32,385 for training and 5,595 for testing.
- Dataset and Evaluation: Evaluation measures spatial alignment with OPR, RTA, and SLE, while FID, FAED, IS, and CLIP Score assess image quality.OPR and RTA measure sector-level correctness, whereas SLE measures angular distance from the target BFoV center.
- Quantitative Results: 98.59 OPR and 36.91 RTA are PanoCtrl’s best spatial-alignment scores, surpassing HunyuanWorld by 13.26 and DiT360 by 9.23, respectively.PanoCtrl also reduces SLE to 25.34 versus 47.63 for the strongest competing method.
- Quantitative Results: PanoCtrl achieves the best FID of 46.86, best FAED of 2.77, and highest CLIP score of 32.54 among the compared methods.Its IS is 3.35, slightly below DiT360’s 3.37, with the reported gap described as marginal.
- Qualitative Results: Qualitative comparisons show more accurate placement within designated BFoVs, clearer spatial separation, coherent layouts, and more realistic visual consistency than representative baselines.The comparisons focus on challenging scenes containing multiple objects and directional descriptions.
4.3 Ablation Study
Ablations show that PanoParse, object-aware attention, and spatial residual enhancement contribute complementary benefits. The studies also identify balanced settings for spatial blending and object-query capacity.
- Main Components: Adding PanoParse improves OPR and RTA, showing that explicit object-level spherical parsing provides structural guidance for controllable generation.The full combination of components achieves the best results across all metrics.
- Main Components: Spatial residual enhancement improves geometric accuracy and reduces SLE, but image quality is slightly affected without semantic interaction.Object-aware attention instead improves both alignment and perceptual quality.
- Training Objectives: Removing parsing loss L_parse degrades OPR, RTA, and SLE, while removing object-level loss L_obj especially harms local alignment and semantic consistency.The full model requires both objectives to jointly optimize layout prediction and generation.
- Design Choices: The best overall performance occurs at spatial blending coefficient α = 0.7, balancing stronger spatial control against global image coherence.Larger α values slightly hurt alignment and quality because overly strong spatial constraints interfere with generation consistency.
- Design Choices: Eight object queries provide the best spatial performance and a balance between representation capacity and generation stability.Sixteen queries yield only marginal FID and CS gains without further alignment improvement.
5 Conclusion
PanoCtrl bridges natural language and spherical panoramic space through structured object-level conditions integrated into diffusion generation. With PanoGround and extensive experiments, it achieves state-of-the-art spatial alignment and image quality.
- PanoCtrl converts textual descriptions into structured object-level spherical conditions and integrates them into the diffusion process.
- PanoGround provides object-level spherical annotations and directional descriptions for controllable panoramic generation.
- PanoCtrl achieves state-of-the-art performance in both spatial alignment and image quality.