PaperlayerView exact revision on arXiv

Source-linked AI summary

Panoramic Affordance Prediction

Zixin Zhang, Chenfei Liao, Hongfei Zhang, Harold Haodong Chen, Kanghao Chen, Zichen Wen, Litao Guo, Bin Ren, Xu Zheng, Yinchuan Li, Xuming Hu, Nicu Sebe, Ying-Cong Chen

arXiv:2603.15558v1cs.CVcs.RO

TL;DR

Affordance prediction is largely limited to pinhole-camera views that fragment environmental context. The paper introduces panoramic affordance prediction through PAP-12K and a training-free fovea-inspired PAP pipeline, which significantly outperforms existing methods on the benchmark.

  • Problem

    Existing affordance research is predominantly confined to pinhole cameras, whose restricted FoV fragments observations and can miss peripheral or rear interaction targets.

  • Method

    The paper introduces PAP-12K and PAP, a training-free coarse-to-fine pipeline using recursive visual routing, adaptive gaze, and cascaded affordance grounding.

  • Results

    PAP significantly outperforms existing state-of-the-art methods on PAP-12K, achieving 71.56% gIoU and 62.30% cIoU.

  • Takeaways & Limitations

    The results highlight the potential of panoramic perception for connecting holistic 360° scene understanding with actionable intelligence in embodied AI.

Abstract

from arXiv · show

Affordance prediction serves as a critical bridge between perception and action in embodied AI. However, existing research is confined to pinhole camera models, which suffer from narrow Fields of View (FoV) and fragmented observations, often missing critical holistic environmental context. In this paper, we present the first exploration into Panoramic Affordance Prediction, utilizing 360-degree imagery to capture global spatial relationships and holistic scene understanding. To facilitate this novel task, we first introduce PAP-12K, a large-scale benchmark dataset containing over 1,000 ultra-high-resolution (12k, 11904 x 5952) panoramic images with over 12k carefully annotated QA pairs and affordance masks. Furthermore, we propose PAP, a training-free, coarse-to-fine pipeline inspired by the human foveal visual system to tackle the ultra-high resolution and severe distortion inherent in panoramic images. PAP employs recursive visual routing via grid prompting to progressively locate targets, applies an adaptive gaze mechanism to rectify local geometric distortions, and utilizes a cascaded grounding pipeline to extract precise instance-level masks. Experimental results on PAP-12K reveal that existing affordance prediction methods designed for standard perspective images suffer severe performance degradation and fail due to the unique challenges of panoramic vision. In contrast, PAP framework effectively overcomes these obstacles, significantly outperforming state-of-the-art baselines and highlighting the immense potential of panoramic perception for robust embodied intelligence.

1 Introduction

The paper introduces panoramic affordance prediction to address the restricted, fragmented observations of pinhole cameras, then presents PAP-12K and a training-free PAP pipeline for panoramic imagery.

  • Motivation: Affordance prediction connects visual perception with actionable capabilities by identifying where and how agents can interact with environments.It supports downstream applications including task planning, tool usage, and object manipulation.
  • Motivation: Pinhole-camera affordance methods provide restricted FoV, fragment observations, and can miss peripheral or rear interaction targets.Robots may need repeated reorientation, increasing time and computational memory burdens.
  • Motivation: Panoramic cameras capture a 360° FoV in one shot, preserving global spatial relationships and enabling holistic scene understanding.The paper identifies panoramic affordance reasoning as a largely unexplored extension of panoramic vision.
  • Contributions: PAP-12K is a large-scale benchmark with over 1,000 12K images, more than 12k QA pairs, affordance masks, and annotations for panoramic challenges.The dataset uses 11904×5952 imagery across hundreds of scenes and 12 categories, including geometric distortion, scale variation, and boundary discontinuity.
  • Contributions: PAP is a training-free, fovea-inspired coarse-to-fine pipeline combining Recursive Visual Routing, Adaptive Gaze, and Cascaded Affordance Grounding.The stages progressively localize targets, rectify local distortions, and extract precise instance-level affordance masks.
  • Contributions: PAP achieves state-of-the-art performance on PAP-12K and overcomes panoramic-image challenges without specialized panoramic fine-tuning.The paper reports that perspective-image baselines suffer severe degradation and largely fail when applied to panoramic environments.

2 Related Work

Related work covers affordance prediction and panoramic vision separately, while emphasizing that affordance reasoning has rarely used the holistic context provided by panoramic imagery.

  • Affordance Prediction: Affordance prediction supports embodied tasks by bridging perception and manipulation for planning, object interaction, and tool usage.It has been integrated into numerous robotic systems.
  • Affordance Prediction: Existing affordance research predominantly uses single-frame pinhole-camera imagery, producing fragmented views and missing spatial context or interaction targets outside the current view.This limits applicability in complex real-world scenarios.
  • Panoramic Vision: Panoramic images provide a 360° field of view that preserves complete spatial context and global geometric relationships.Distortion and scale variation remain processing challenges.
  • Research Gap: Although panoramic vision is used for embodied scene understanding, its potential for affordance reasoning remains largely unexplored.The paper positions holistic panoramic context as a way to represent action possibilities across the environment.
  • Dataset Comparison: Table 1 compares PAP-12K with existing affordance and panoramic datasets, using the largest resolution when a dataset has multiple resolutions.The comparison concerns dataset resolution and category type as described by the caption.

3 PAP-12K

PAP-12K is a large-scale benchmark for panoramic affordance prediction, combining ultra-high-resolution 360° imagery with reasoning-based questions and precise affordance masks. Its design explicitly captures geometric distortion, extreme scale variation, and boundary discontinuity.

  • Dataset Overview: PAP-12K is the first large-scale benchmark dedicated to panoramic affordance prediction.It is designed around the distinctive challenges of 360° ERP imagery.
  • Annotations: Its QA pairs require complex reasoning and are paired with precise pixel-level segmentation masks for explicit affordance localization.The annotations move beyond simple classification toward logical deduction and spatial grounding.
  • Panoramic-Specific Challenges: PAP-12K targets three panoramic challenges: geometric distortion, extreme scale variations, and boundary discontinuity.Examples include stretched objects, minute targets, and affordance regions wrapping across image boundaries.
  • Dataset Statistics: The dataset contains 1,003 ultra-high-resolution panoramic images, 6,103 annotated object instances, and 13,493 affordance-related questions.Images are captured at 11904 × 5952 resolution across 12 indoor scene types.

4 PAP

PAP is a training-free, fovea-inspired coarse-to-fine pipeline for locating and grounding affordances in ultra-high-resolution panoramic images. It combines grid-based recursive routing, adaptive spherical-to-perspective projection, and cascaded instance grounding.

  • Framework Overview: PAP addresses ultra-high resolution and ERP distortion by mimicking human visual scanning from broad localization to detailed grounding.The pipeline uses peripheral-style search before focused target processing.
  • Recursive Visual Routing via Grid Prompting: Visual Grid Prompting converts continuous target localization into a discrete grid-selection task while producing an explicit object description.The VLM overlays a 4 × 3 numbered grid and selects the target-containing cell.
  • Recursive Visual Routing via Grid Prompting: Recursive Visual Routing repeatedly crops and re-grids sub-scale targets, progressively zooming in while adjusting image downsampling for efficiency.This filters irrelevant panorama content and provides a reliable coarse region for later modules.
  • Adaptive Gaze: Adaptive Gaze projects the localized spherical region onto a tangent plane, producing a target-specific perspective image that reduces ERP distortion and scale variation.The projection also reconstructs objects split across ERP boundaries and avoids panoramic-specific fine-tuning.
  • Cascaded Grounding: A cascaded grounding pipeline applies an open-vocabulary detector and segmentation model to the refined perspective crop for precise masks.The detector uses the explicit object description to generate a bounding box and key points that guide dense segmentation.

5 Experiments

Experiments compare PAP with perspective-image affordance and open-vocabulary segmentation baselines on PAP-12K. PAP leads across the reported metrics, while ablations show that visual grids, recursive routing, and adaptive gaze address distinct panoramic difficulties.

  • Experimental Settings: PAP is evaluated against perspective affordance methods including A4-Agent, Affordance-R1, and AffordanceVLM, plus general open-vocabulary segmentation models.The comparison reflects the lack of direct panoramic affordance baselines.
  • Main Results: PAP achieves 71.56% gIoU and 62.30% cIoU, exceeding A4-Agent by 9.01% and 12.33% absolute margins, respectively.It also reaches 75.49% P50 and 64.97% P50−95.
  • Main Results: PAP remains effective on hard cases involving extreme object sizes, boundary splits, and severe distortion, whereas other methods struggle.The Hard subset contains objects occupying more than 30% or less than 0.1% of the image, or crossing horizontal boundaries.
  • Ablation Studies: Replacing visual grids with textual spatial descriptions causes a drastic drop across all metrics, showing the importance of concrete visual reference points.Chain-of-Thought reasoning helps, but explicit grid overlays are reported as more crucial.
  • Ablation Studies: Recursive Visual Routing improves Hard-subset cIoU and gIoU by 12.30% and 5.03%, compared with 4.50% and 0.78% on Normal cases.The larger Hard-subset gains support iterative zooming for extreme scale variation.
  • Ablation Studies: Removing Adaptive Gaze causes a significant accuracy decline because direct ERP crops preserve spatial distortion and create a domain shift for 2D foundation models.Adaptive Gaze aligns panoramic crops with pretrained 2D model priors without training.

6 Conclusion

The paper introduces panoramic affordance prediction through the PAP-12K benchmark and the training-free PAP framework. Experiments report strong performance, particularly on challenging panoramic cases, motivating further work combining panoramic vision with embodied intelligence.

  • Conclusion: The paper presents the first exploration of Panoramic Affordance Prediction, connecting holistic 360° scene understanding with actionable embodied intelligence.The work frames panoramic perception as a route toward richer affordance reasoning.
  • Conclusion: PAP-12K contributes over 1,000 native 12K panoramic images, more than 12K reasoning-based QA pairs, and precise affordance masks.The dataset is intended to support research on panoramic affordance prediction.
  • Conclusion: PAP is a training-free framework inspired by the human foveal visual system for addressing panoramic vision challenges.The conclusion identifies the framework and benchmark as foundations for future research.
  • Conclusion: Extensive experiments show that PAP significantly outperforms existing state-of-the-art methods, especially in highly challenging scenarios.The authors position the dataset and framework as stepping stones toward combining panoramic vision and embodied intelligence.

A.1 Superparameter Analysis of Visual Prompt

The visual-prompt analysis finds that line-based grids outperform colored blocks, while moderate line widths, font sizes, and grid resolutions provide the most reliable performance. Excessively thick, large, coarse, or dense prompts impair spatial guidance and visual detail.

  • Grid Prompting Style: Line-based grid prompts achieve the best performance across all metrics, reaching a gIoU of 72.69 and cIoU of 63.85.Color-block prompts significantly degrade performance.
  • Experimental Comparisons: Tables and figures compare visual prompt types, line widths, font sizes, and grid resolutions, with red marking the best results where specified.The table captions identify the default settings used in the final pipeline.
  • Grid Line Width: The pipeline remains robust to line width, with 1-pixel lines yielding gIoU 71.93 and cIoU 60.35, while 50-pixel lines severely reduce performance.The best results occur at a moderate 5-pixel width; very thick lines fragment the scene and occlude fine-grained information.
  • Font Size of Grid Coordinates: Font sizes from 10 to 100 produce stable results, with size 50 achieving the best balance before excessively large text harms performance.Large coordinate text can negatively affect visual interpretation, as illustrated in the accompanying visualization.
  • Analysis of Grid Resolution: Grid resolutions of 3 × 3, 4 × 3, 4 × 4, and 5 × 5 perform comparably, whereas 2 × 2 is too coarse and dense grids reduce performance.The analysis identifies a moderate resolution range as preferable for spatial constraints.

A.2 Analysis of Different VLM backbones

PAP’s performance increases with the capability of its VLM backbone, while the pipeline remains flexible enough to trade absolute performance against inference cost by selecting different models.

  • Backbone Capability: Stronger VLM backbones improve PAP’s overall performance metrics, while smaller open-source models produce corresponding declines.Replacing Qwen-3-VL-32B with Gemini-3-Flash substantially improves all metrics; Qwen-3-VL-8B/4B and Qwen-2.5-VL-7B reduce performance.
  • Experimental Setup: Table 10 reports PAP’s performance across VLM models and identifies the default backbone setting used in the final pipeline.The comparison covers models with varying capabilities and scales.
  • Practical Flexibility: PAP can integrate closed-source models for higher absolute performance or lightweight models for a balance between performance and inference cost.This flexibility also allows the pipeline to function as an offline data-generation engine.

A.3 Analysis of Different Resolution

Resolution experiments show that PAP benefits substantially from increasing image resolution from 1000 × 500 to 2000 × 1000, while further scaling to 4000 × 2000 yields only marginal and mixed gains. The decoupled pipeline and future end-to-end direction frame this trade-off within broader deployment and modeling considerations.

  • Image Resolution: 2000 × 1000 produces a significant performance boost over 1000 × 500, whose low resolution limits fine-grained detail capture.The 2000 × 1000 setting is visibly sharper and supports better detail extraction.
  • Image Resolution: 4000 × 2000 delivers only marginal gains over 2000 × 1000, with slight improvements in gIoU, P50, and P50−95 but lower cIoU.The authors attribute this pattern to Recursive Visual Routing’s coarse-to-fine use of global context and localized detail.
  • Decoupled Pipeline: PAP’s decoupled reasoning and grounding paradigm supports flexible VLM integration, zero-shot generalization, and intermediate-step interpretability.The approach is contrasted with end-to-end solutions on flexibility and model specialization.
  • Future Direction: The authors identify native ERP understanding and efficient end-to-end affordance inference as valuable directions for future work.PAP can serve as a zero-shot data-generation engine for scaling training data toward such models.

B.3 Managing Cascading Errors

PAP addresses cascading errors from grid-boundary localization by adding redundant margin to the adaptive-gaze field of view. This prevents boundary-crossing objects from being split before grounding.

  • Error Source: Decoupled pipelines can accumulate errors when Recursive Visual Routing selects only one grid for an object crossing a grid boundary.A narrowly selected FoV may then split the object in half and compromise subsequent grounding.
  • Error Source: Recursive Visual Routing usually outputs both grids containing a boundary-crossing object, but occasionally returns only one.The single-grid case is the failure mode targeted by the adaptive-gaze redundancy.
  • Error Mitigation: Figure 12 visualizes routing errors and grid coverage, showing the adaptive-gaze module covering the object-containing grid with a small redundant margin.The examples include a coffee table and a curtain near grid boundaries.
  • Error Mitigation: PAP adds an adaptive-gaze FoV margin of around 10 degrees so boundary-crossing objects remain covered in the normal view.The redundancy extends coverage slightly beyond the selected grid.

C More Implementation Details

The implementation details specify the prompts, geometric transformations, and baseline adaptations used to process panoramic affordance inputs.

  • C.1 System Prompt of Recursive Visual Routing: The Recursive Visual Routing prompt overlays a 4×3 grid and asks the VLM to identify target grids, object name, task, object part, and small-object status.The output schema records grid indices from 1 to 12 alongside the target description and whether refinement is needed.
  • C.2 Formulation of the Adaptive Gaze Module: Adaptive Gaze constructs a rectilinear viewport by mapping target pixels to 3D rays, rotating them, converting them to spherical coordinates, and sampling ERP coordinates.Bilinear interpolation then produces the final rectilinear image from the discrete ERP grid.
  • C.2 Formulation of the Adaptive Gaze Module: The virtual-camera focal length is computed from image width and horizontal FoV before defining each target-image pixel’s local 3D ray.The optical center is typically taken as the image center.
  • C.2 Formulation of the Adaptive Gaze Module: The rotated rays are normalized and mapped to longitude-latitude coordinates, then converted into ERP pixel coordinates using the panorama dimensions.Longitude spans [−π, π] and latitude spans [−π/2, π/2].
  • C.3 Details of the Implementation of Baseline Methods: Perspective-image baselines are adapted by resizing panoramic inputs, because direct full-resolution processing either nearly fails or incurs prohibitive computation.A4-Agent inputs are resized to 2000 × 1000, while other methods are matched to their training resolutions.

D More Details about the Data Annotation Process

The annotation process combines automated question generation, multi-view inspection, manual mask creation, and repeated verification to produce unambiguous affordance annotations across PAP-12K scenes.

  • Phase 1: Affordance Question Formulation: Question formulation uses batch generation by an AI agent followed by manual human refinement and filtering.The process is designed for scalable production while retaining expert review.
  • Phase 1: Affordance Question Formulation: The generation rules express affordances as human needs, require unique solutions, and distinguish similar functional roles such as surfaces versus tools or containers versus contents.The prompt also accounts for ERP pole distortion by providing both the original panorama and cubemap projections.
  • Phase 1: Affordance Question Formulation: The agent generates 2 to 4 distinct questions per target object to represent different functional uses.Question variety is treated as a separate design requirement.
  • Phase 1: Affordance Question Formulation: Gemini-3-Pro and GPT-5 independently generate batches whose outputs are merged, screened, and refined by human experts.This combines multiple model outputs with expert quality control.
  • Phase 1: Affordance Question Formulation: Annotators jointly inspect one ERP panorama and six cubemap faces to identify objects and generate affordance VQA pairs with unique target answers.The six faces are ordered Front, Right, Back, Left, Top, and Bottom.
  • Phase 1: Affordance Question Formulation: Object descriptions remain concise for unique objects but add location, appearance, or state when similar objects require disambiguation.The annotation rules prioritize specificity only when necessary.
  • Phase 2: Mask Segmentation: Five annotators manually create segmentation masks in a customized WebUI after affordance question-answer pairs are established.The interface displays relevant annotation information to support mask creation.
  • Phase 3: Final Verification: Two complete manual reviews remove over 1,000 ambiguous instances from an initial pool exceeding 15,000 samples.The remaining dataset undergoes final verification of questions, answers, and segmentation masks.
Loading 2603.15558v1…