PaperlayerView exact revision on arXiv

Source-linked AI summary

PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection

Sara Pieri, Evangelos Kazakos, Shizhe Chen, Josef Sivic, Cordelia Schmid

arXiv:2609.19143v1cs.CVcs.CL

TL;DR

Panoptic grounded captioning requires complete scene descriptions with pixel-level grounding for every referring phrase, a capability existing methods often handle incompletely or inaccurately. The paper introduces PanoCaps and PANORAMA, whose proposal-selection approach achieves the best overall grounding on PanoCaps while retaining detailed, mask-consistent captions.

  • Problem

    Existing vision-language systems struggle to associate detailed, comprehensive scene descriptions with accurate pixel-level masks.

  • Method

    The paper introduces PanoCaps and PANORAMA, which grounds each phrase by selecting relevant masks from a phrase-conditioned proposal pool generated by a pretrained segmenter.

  • Results

    PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks.

  • Takeaways & Limitations

    PanoCaps and PANORAMA support detailed scene descriptions with precise entity-level, mask-consistent grounding.

  • Takeaways & Limitations

    The work focuses on image-level grounding and does not yet cover region description or video grounding.

Abstract

from arXiv · show

Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably associating them with image pixels remains challenging. Existing methods that combine dense captioning with pixel-level grounding often produce either incomplete descriptions or inaccurate segmentation masks. We study this problem through panoptic grounded captioning, a task that requires a VLM to describe both foreground objects and background regions while grounding each referring phrase with pixel-level masks. We make three contributions. First, we introduce PanoCaps, a human-annotated benchmark constructed from panoptic segmentation datasets. It provides dense captions with near-complete pixel coverage and image-text alignments at the entity level, supporting both training and evaluation. We further propose a phrase-mask matching protocol and a generalized Panoptic Quality (gPQ) metric that jointly evaluates textual and mask agreement. Second, we formulate phrase grounding as selection from a phrase-conditioned pool of mask proposals and introduce PANORAMA, a VLM that conditions a pretrained segmenter on contextualized phrase representations to obtain candidate masks and learns to select those corresponding to each phrase. Training this interface jointly with caption generation enables PANORAMA to produce high-quality masks while allowing each phrase to refer to a single region or multiple instances. Third, PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks. Experiments show that our method produces precise entity-level segmentations while maintaining detailed, mask-consistent captions. Code, data and models are available at https://www.di.ens.fr/willow/research/panorama/.

1 Introduction

Panoptic grounded captioning seeks complete scene descriptions whose referring phrases are precisely aligned with pixels, addressing omissions, coarse masks, and noisy grounding in existing systems. The paper introduces PanoCaps and PANORAMA, achieving strong grounding while preserving detailed, mask-consistent language.

  • Existing methods often omit entities, produce incomplete descriptions, or ground phrases with coarse and imprecise masks.These shortcomings limit comprehensive and spatially precise scene understanding for systems acting in the physical world.
  • PanoCaps combines human-written, full-scene captions with near-complete pixel coverage and fine-grained phrase-mask alignments.The benchmark addresses the trade-off between dense but noisy automatic supervision and sparse human annotations.
  • PANORAMA grounds each phrase by selecting from a phrase-conditioned mask-proposal pool generated by a pretrained segmenter.A contextualized phrase representation conditions candidate masks, while a learned scorer selects the relevant proposals.
  • PANORAMA achieves the best overall grounding on PanoCaps and matches or exceeds specialized models across several pixel-level grounding tasks.The reported results indicate detailed, mask-consistent language generation without sacrificing localization quality.
  • The benchmark adds a phrase-mask matching protocol and generalized Panoptic Quality metric for jointly evaluating textual and mask agreement.These components extend evaluation beyond caption quality or segmentation accuracy considered separately.

2 Related Work

Prior grounding methods vary in their spatial representations and often provide sparse, noisy, or limited supervision. PanoCaps is positioned as a denser, more precise resource supporting free-form descriptions and explicit phrase-region alignment.

  • 2.1 Spatial Grounding Methods: Spatial grounding methods differ in whether they use coordinates, spatial tokens, hybrid representations, or segmentation-based outputs.Bounding-box approaches provide region-level rather than pixel-level grounding.
  • 2.2 Grounded Captioning Datasets: Existing grounded-captioning sources can be sparse or noisy, leaving regions ungrounded or assigning incorrect generic descriptions.GranD-f grounds only a few objects, whereas COCONut-PanCap is denser but contains incorrect references and generic statements.
  • 2.2 Grounded Captioning Datasets: PanoCaps provides dense, precise grounding with full-scene human-written captions and explicit phrase-to-region links.Its examples support multi-referencing, multi-region grounding, and free-form descriptions.
  • 2.1 Spatial Grounding Methods: PANORAMA conditions its candidate mask pool on contextualized phrase representations rather than selecting from an image-level, phrase-independent pool.This design leverages a pretrained segmentation model and supports phrases referring to multiple regions.

3 The PanoCaps Benchmark

PanoCaps is a human-annotated benchmark designed for comprehensive, pixel-level scene understanding, combining full-scene captions with verified phrase-mask alignments. It also defines evaluation procedures that jointly assess textual and spatial agreement.

  • 3.1 Annotation Pipeline: Human annotators write free-form captions and link every groundable phrase to one or more segmentation masks, followed by independent verification.The annotation process targets linguistic quality, completeness of region coverage, and alignment correctness.
  • 3.2 Benchmark Features: PanoCaps contains 3.5K images, approximately 34K panoptic masks, and 31.3K grounded entities, with roughly nine entities per image.Its annotations span diverse image sources and include substantial lexical diversity.
  • 3.2 Benchmark Features: Approximately 99% of image pixels are covered by grounded regions spanning both foreground objects and background regions.This near-complete coverage makes the annotations dense rather than sparse for panoptic grounded captioning.
  • 3.2 Benchmark Features: PanoCaps supports flexible references: 9.8% of phrases refer to several masks, while 12.5% of masks are referenced by more than one phrase.These patterns reflect multi-instance and overlapping human descriptions of scenes.
  • 3.3 Evaluation: Evaluation measures caption quality, segmentation accuracy, and grounding quality, with grounding requiring both correct description and localization.The benchmark introduces open-text phrase-mask matching using textual and mask similarities, including Hungarian matching.

evaluation.

The generalized Panoptic Quality metric jointly evaluates phrase similarity and mask overlap while penalizing unmatched predictions and ground-truth regions.

  • gPQ combines graded textual similarity and mask IoU for matched phrase-region pairs, while penalizing unmatched predictions and ground-truth masks.For fixed semantic categories, textual similarity becomes a binary same-class indicator, reducing gPQ to class-agnostic PQ pooled over segments.
  • The metric uses matched-pair agreement together with false positives and false negatives to assess both caption grounding quality and coverage.Its formulation includes Simmask, Simtext, FP, and FN, with both similarity terms bounded in [0, 1].

4 Methodology

PANORAMA grounds generated phrases by conditioning mask proposals on phrase representations and selecting the matching candidates, with joint objectives supporting flexible multi-region predictions.

  • 4.2 The Proposed Model: PANORAMA: PANORAMA projects each phrase’s [SEG] hidden state into a concept vector that conditions a pretrained segmenter to generate phrase-specific candidate masks.The proposal pool is referent-conditioned rather than a fixed, class-agnostic set.
  • 4.2 The Proposed Model: PANORAMA: A learned match scorer compares each proposal query with the concept vector and selects candidates whose scores exceed an inference threshold.The scorer is trained jointly with caption generation, encouraging higher similarity for proposals corresponding to the intended referent.
  • 4.2 The Proposed Model: PANORAMA: Independent proposal thresholding supports zero, one, or multiple selected masks per phrase; selected instances are retained separately or united for binary evaluation.Expressions with no target region are handled by emitting no [SEG] token.
  • 4.3 Training and Inference: Training matches phrase target regions to proposals with Hungarian assignment, combining mask Dice overlap and match scores to create selection labels.Mask overlap is primary, while the score helps distinguish proposals with similar overlap.
  • 4.3 Training and Inference: The model jointly optimizes text generation, proposal selection, proposal-mask, and semantic-mask losses.The text loss supervises captions, grounding delimiters, and [SEG] markers; mask losses train matched proposals and the semantic prediction.
  • PANORAMA unifies grounding settings across generated versus provided expressions and single-region versus multi-region targets, with PanoCaps combining both axes.This formulation connects grounded captioning to referring, generalized referring, and category-level grounding tasks.

5 Experiments

PANORAMA’s proposal-selection design delivers strong panoptic grounding and transfers across grounded captioning, referring segmentation, and generalized grounding tasks. Ablations and data comparisons show that phrase-conditioned proposals, contextualized VLM representations, and dense human-verified supervision are central to performance.

  • 5.2 Ablation Studies: Direct mask generation costs 7.5 gPQ and 20.0 AP50 on PanoCaps, whereas removing dense semantic supervision reduces average RefCOCO cIoU by 2.5 points.The ablations indicate that proposal selection is especially important for dense grounding, while semantic supervision benefits single-entity grounding.
  • 5.2 Ablation Studies: Removing phrase conditioning costs 1.8 gPQ and 2.6 Recall on PanoCaps and 2.9 RefCOCO cIoU, showing the value of phrase-specific proposal pools.Conditioning steers proposal generation toward the entity named by each phrase.
  • 5.2 Ablation Studies: PanoCaps improves specialized SAMTok-8B from 19.7 to 44.5 gPQ, while finetuning across architectures raises gPQ by 15–25 points.The benchmark uses far fewer images than competing sources but provides human-written, fully verified captions with dense phrase-mask alignments.
  • 5.3.1 Main Results: PANORAMA-4B achieves the best results on five of seven reported metrics, including the best gPQ of 45.6 versus 44.5 for the strongest finetuned baseline.Its advantage comes from more accurate pixel-level grounding at comparable recall, while finetuned baselines retain an edge on text-only CAPTURE and SAMTok marginally leads grounding recall.
  • 5.3.2 Additional Results: After finetuning, PANORAMA-4B obtains the best result on seven of ten GCG metrics, including every grounding metric on both validation and test splits.SAMTok-7B remains ahead on several caption-quality metrics, but PanoCaps-trained grounding transfers to shorter, sparser descriptions.
  • 5.3.2 Additional Results: On GroundingSuite, PANORAMA ranks first in every category and improves overall gIoU from 67.8 to 75.5 without benchmark-specific training.Its architecture supports multi-region grounding, with the largest gains in Part and Multi categories.

6 Conclusion

The paper presents PanoCaps and PANORAMA as complementary contributions for dense, spatially grounded scene understanding. PanoCaps supplies human-verified caption-mask supervision, while PANORAMA uses phrase-conditioned mask proposal selection to connect language generation with segmentation.

  • 6 Conclusion: PanoCaps combines human-written captions, explicit phrase-mask alignments, and near-complete pixel coverage for panoptic grounded captioning.The dataset is constructed from established panoptic segmentation corpora and supports both training and evaluation.
  • 6 Conclusion: PANORAMA grounds each referring phrase by conditioning a pretrained proposal model on a VLM concept vector and selecting relevant candidate masks.The interface separates referent identification from boundary prediction and supports singular or plural references.
  • 6 Conclusion: PanoCaps is designed to pair comprehensive scene descriptions with precise spatial delineation, addressing the annotation trade-off targeted by the paper.Its construction emphasizes dense grounding while retaining fluent, detailed captions.
  • A.2 Annotation Protocol: The annotation protocol filters for reliable, interpretable, diverse, and high-coverage images before human captioning and phrase-level mask alignment.Annotators use color-coded panoptic overlays and label maps, while a separate verification stage checks compliance, alignment, and scene coverage.

A.3 Additional Dataset Details

Additional statistics characterize PanoCaps as dense, lexically varied, and structurally diverse across its dataset splits. The corpus combines broad scene composition with near-complete mask coverage and varied region scales.

  • A.3 Additional Dataset Details: Approximately 99% pixel coverage and an average of nine grounded entities per image indicate dense annotations across PanoCaps splits.Coverage is consistent across splits, suggesting comparable annotation density.
  • A.3 Additional Dataset Details: PanoCaps combines ADE20K, COCONut, and VIPSeg while keeping source training and evaluation images in separate splits.The sources contribute complementary visual domains and scene contexts.
  • A.3 Additional Dataset Details: The dataset exhibits substantial lexical diversity through varied free-form noun phrases beyond the source segmentation taxonomies.Table 12 summarizes caption length and unique noun-phrase counts by split.
  • A.3 Additional Dataset Details: Its region distribution spans things and stuff masks across small, medium, and large spatial scales.This mixture reflects the structural complexity relevant to grounding and caption generation.
  • A.3 Additional Dataset Details: Overall, PanoCaps combines dense pixel supervision, diverse scenes, and lexically diverse free-form annotations.These properties are summarized across the split-level statistics and qualitative examples.

A.4 Comparison with COCONut-PanCap

Compared on 1,550 shared images and masks, PanoCaps provides denser, broader, and more lexically varied annotations than COCONut-PanCap. Its preprocessing also improves the consistency of training captions and phrase-mask parsing.

  • A.4 Comparison with COCONut-PanCap: PanoCaps uses 21% more unique labels, reflecting richer lexical diversity on the shared-image comparison.The metric counts distinct free-form phrases used to describe masks.
  • A.4 Comparison with COCONut-PanCap: 50% higher pixel coverage shows that PanoCaps descriptions capture a larger portion of the visual scene than COCONut-PanCap.The comparison uses the same images and segmentation masks.
  • A.4 Comparison with COCONut-PanCap: PanoCaps references 26% more available masks and includes 23% more grounded masks per image than COCONut-PanCap.The per-image count includes only masks explicitly referenced in captions.
  • A.4 Comparison with COCONut-PanCap: Regenerating noisy COCONut-PanCap captions produces more consistent phrase-level grounding for the training mixture.The cleaned captions address malformed or invalid mask references, while PanoCaps remains denser and more specific.
  • A.4 Comparison with COCONut-PanCap: Overall, PanoCaps improves referenced-mask coverage, pixel coverage, and vocabulary variation relative to COCONut-PanCap.The comparison is conducted on overlapping images with identical segmentation masks.

B.1 Evaluation Protocol

The evaluation protocol measures captioning, segmentation, and grounding quality using open-text phrase-mask matching. It combines lexical and embedding-based phrase similarity with spatial agreement and graded panoptic quality.

  • B.1 Evaluation Protocol: Models generate captions with phrase tags followed by [SEG] tokens whose hidden states condition proposal masks for grounding.A learned scorer selects candidate masks corresponding to each phrase.
  • B.1 Evaluation Protocol: PanoCaps evaluation covers caption quality with CAPTURE, segmentation with AP50 and mIoU, and grounding through phrase-mask matching.The three axes separately assess textual description, mask accuracy, and phrase-region association.
  • B.1 Evaluation Protocol: Phrase similarity combines exact matches, WordNet synsets for single-token phrases, and Sentence-BERT scores in [0, 1].Masks with IoU ≥0.9 and text similarity ≥0.5 are merged before matching.
  • B.1 Evaluation Protocol: One-to-one Hungarian matching assigns pairs using the mean of text and mask similarities, with correctness requiring both similarities to reach 0.5.Precision, Recall, and F1 are computed from the resulting correct matches.
  • B.1 Evaluation Protocol: Generalized Panoptic Quality preserves graded textual and spatial agreement instead of reducing evaluation to binary matches.This complements the binary Precision, Recall, and F1 grounding metrics.

B.2 Comparison with Prior GCG Evaluation

The paper replaces greedy mask-IoU matching with global assignment that jointly considers textual and spatial similarity, exposing errors missed by prior GCG evaluation.

  • Global assignment on mean textual and spatial similarity resolves ambiguous phrase-mask correspondences jointly rather than in discovery order.This differs from greedy mask-IoU matching gated by a soft text-similarity check.
  • 98.8% of BiRD pairs exceed the prior BERT similarity threshold, including unrelated pairs, whereas the proposed measure separates low- and high-relatedness pairs at 0.26 versus 0.67.The proposed 0.5 threshold therefore becomes a meaningful operating point, unlike the prior scorer’s threshold.
  • A plant-pot prediction with an accurate mask but an unrelated phrase passes the prior BERT gate yet is rejected by the proposed metric.The example demonstrates why spatial accuracy alone does not establish a correct phrase-mask match.
  • Two hallucinated predictions leave prior matched-pair metrics perfect, while Precision and gPQ expose them.The example shows that the reported metrics can penalize predictions omitted by prior matched-pair evaluation.

C Evaluation of Generalist VLMs on PanoCaps

The paper evaluates Gemini 2.5 Pro and Qwen3-VL-235B-A22B on PanoCaps with a two-stage protocol because neither natively produces full captions and pixel-level masks together.

  • The two-stage protocol first generates a caption with interleaved entity tags, then grounds each parsed entity spatially.Captions are returned as JSON, and grouped tags can represent multiple instances of the same entity type.
  • Gemini 2.5 Pro receives prompts for JSON outputs containing bounding boxes, segmentation masks, and text labels for each entity.The model is evaluated using direct segmentation outputs in the second stage.
  • Qwen3-VL-235B-A22B receives box-and-label prompts, after which SAM derives segmentation masks from its predicted boxes.This accommodates Qwen3-VL-235B-A22B’s box-level grounding limitation.
  • Model-specific prompt adjustments address instruction-following and formatting differences, including few-shot examples for Qwen3-VL-235B-A22B.Failed or unparsable calls are retried up to five times before recording no prediction.

D Additional Results

Additional experiments examine proposal-model training, inference-time selection, efficiency, and prompt templates or examples used for captioning and grounding.

  • Unfreezing the DETR decoder and mask heads adds roughly 10% training time without measurable gains on any metric.The default instead unfreezes only the fusion encoder and match scorer, with a small recall gain from fully freezing the proposal model.
  • 0.5 is the best inference-time selection threshold on gPQ in every seed, while fixed proposal counts degrade performance sharply.All selection-rule variants reuse the same three trained checkpoints.
  • Table 17 reports total and trainable parameters plus median single-A100 latency over 100 test images for each finetuned grounding model.The cost covers the full grounded-captioning pipeline with batch size 1.
  • PanoCaps examples pair original images and panoptic overlays with grounded captions whose phrase colors identify their masks, including multi-instance phrases such as three benches.The examples draw from COCONut, VIPSeg, and ADE20K.
  • Prompt templates specify detailed scene descriptions, entity tags, unique identifiers, and JSON caption outputs for model training and evaluation.Separate templates cover caption generation and segmentation or object detection for the evaluated models.
Loading 2609.19143v1…