Fully Convolutional Networks for Panoptic Segmentation

Motivation

• Panoptic segmentation must assign every pixel a semantic label and unique identity while unifying countable things and uncountable stuff.Things require instance-aware, object-varying features, whereas stuff benefits from semantically consistent representations.Source: §1 · p. 1
• Prior methods commonly separate things and stuff into tailored branches, using boxes or auxiliary predictions for things and pixel-wise FCN predictions for stuff.These designs include box-based and box-free instance streams, with box-free methods still requiring extra predictions and postprocessing.Source: §1 · p. 1

Panoptic FCN

• Panoptic FCN uniformly represents things and stuff with generated kernels in a fully convolutional pipeline.A kernel generator produces instance-specific or category-specific weights, while a feature encoder supplies shared high-resolution features.Source: §1 · pp. 1–2
• Unlike dense prediction followed by NMS, Panoptic FCN generates instance-aware kernels directly for each instance.For stuff, it aggregates global context into semantically consistent kernels and predicts existing semantic classes in a whole-instance manner.Source: §1 · p. 2

Results

• 44.3% PQ on COCO val and 47.5% PQ on COCO test-dev accompany leading results of 61.4% PQ on Cityscapes and 36.9% PQ on Mapillary Vistas.These results use single-scale input and are reported without bells-and-whistles.Source: §1 · p. 2

Panoptic segmentation

• Traditional panoptic systems separately segment things and stuff, then combine predictions from different models, creating computational overhead.Single-model approaches such as Panoptic FPN, AUNet, and UPSNet still use separate branches.Source: §2 · p. 2
• Foreground things are commonly represented with boxes or center offsets, while background stuff is predicted through a parallel FCN branch.Uniform pixel-level affinity approaches do not fully resolve the inherent ambiguity between instances.Source: §2 · p. 2

Instance segmentation

• Box-free instance segmentation avoids object boxes, but prior methods still require instance aggregation or object-level duplicate removal.Panoptic FCN generates one kernel per object and convolves detail-rich features directly without object-level duplicate removal.Source: §2 · p. 2

Semantic segmentation

• Semantic segmentation assigns each pixel a category without distinguishing diverse object identities.Related methods improve semantic consistency by capturing wider contextual cues or modeling long-range pixel relationships.Source: §2 · p. 2

Framework

• Panoptic FCN uses a kernel generator, kernel fusion, and feature encoder in a unified fully convolutional pipeline.The generator creates kernels, fusion merges matching identities across stages, and the encoder produces high-resolution features.Source: §3 · p. 2, Figure 2 · p. 3

Kernel Generator

• The position head represents things by object centers and stuff by semantic regions, treating same-category background as one instance.Convolutional heads predict position maps and category channels for things and stuff separately.Source: §3.1 · p. 3
• The kernel head concatenates relative coordinates with stage features and selects position-specific weights to represent predicted things and stuff.Each selected kernel is a C_e×1×1 weight associated with a predicted category and location.Source: §3.1 · p. 3

Kernel Fusion

• Kernel fusion averages weights assigned the same identity, preserving separate object embeddings for things and category-consistent embeddings for stuff.Object identity uses cosine similarity above a threshold, whereas stuff kernels sharing a category receive one identity.Source: §3.2 · p. 4

Feature Encoder

• The feature encoder generates a high-resolution encoded feature, and M+N kernels produce M+N predictions at W/4×H/4 resolution.The encoded feature incorporates positional cues before direct convolution generates whole-image predictions.Source: §3.3 · p. 4

Training and Inference

• Training uses object centers and stuff-region points to generate kernels, while inference aggregates candidates and follows a generate-kernel-then-segment workflow.Multiple high-scoring points may represent each object, but stuff uses one factor k=1 treatment across points of the same category.Source: §3.4 · p. 4

Experimental Setting

• Panoptic FCN uses ResNet-FPN, with P3–P7 features for kernel generation and P2–P5 features for high-resolution feature generation.The kernel generator is shared across FPN stages and uses GroupNorm, ReLU, and final projection convolutions.Source: §4.1 · p. 5

Component-wise Analysis

• Increasing kernel-generator depth improves performance, while deformable convolutions in the position head add 1.4% PQ, especially for stuff regions.The best kernel-generator setting uses 3 stacked Conv3 × 3 layers with 256 channels.Source: §4.2 · pp. 5–6
• Position cues improve performance by up to 1.4% PQ when attached to both the kernel head and feature encoder.Attaching coordinates to either component alone yields improvements of up to 0.3% PQ.Source: §4.2 · p. 6
• Weighted dice loss improves performance by 1.1% PQ, and extending training from 1× to 2× and 3× adds 1.9% and 0.4% PQ, respectively.The weighted loss samples 7 top-scoring kernels per instance and optimizes them together.Source: §4.2 · p. 7

5. Conclusion

• Panoptic FCN represents things and stuff uniformly through a fully convolutional pipeline.The framework is designed as a unified alternative to separate processing branches.Source: §5 · p. 8
• Kernel generator and kernel fusion generate unique kernel weights for each object instance or semantic category.These components provide the representation used for both foreground instances and background stuff.Source: §5 · p. 8
• Predictions are produced by convolving high-resolution features directly.The feature encoder supplies the high-resolution feature used by the convolutional prediction process.Source: §5 · p. 8

A. Experimental Details

• The section presents technical details of the Panoptic FCN training process.It focuses on how the proposed network is optimized during training.Source: A. Experimental Details · p. 11
• The section presents technical details of the Panoptic FCN inference process.It also describes how predictions are produced during inference.Source: A. Experimental Details · p. 11
• Training and inference are both covered as parts of the proposed Panoptic FCN workflow.The section addresses both stages rather than introducing a separate method.Source: A. Experimental Details · p. 11

A.1. Training Details

• Mass centers generate object coordinates for ground-truth assignment in the position head.Mass centers are reported to be more robust than box centers, especially with weighted dice loss.Source: §A.1 · p. 11
• Weighted dice loss selects k top-scoring positions within each object region to represent the same object with multiple kernels.The selected kernels generate k predictions optimized against the same segmentation target.Source: §A.1 · p. 11
• Stuff regions merge positions sharing a category at each stage using AvgCluster to produce one specific kernel.For stuff regions, the factor k is treated as 1.Source: §A.1 · p. 11
• Table 15 compares weighted dice loss and center type settings on the COCO validation set.The table defines weighted and center type as the loss and ground-truth-generation settings, respectively.Source: Table 15 · p. 11

A.2. Inference Details.

• COCO inference retains 100 top-scoring object kernels and uses a 0.4 threshold for soft-mask binarization.These settings are used for object prediction and converting soft masks to binary results.Source: §A.2 · p. 11
• Cityscapes and Mapillary Vistas inference retain 200 top-scoring object kernels.Both traffic-related datasets use the same retained-kernel count.Source: §A.2 · p. 11
• Cityscapes and Mapillary Vistas use a cosine threshold of 0.95 and a mask threshold of 0.5.These thresholds are specified for inference on both datasets.Source: §A.2 · p. 11

• The qualitative evaluation covers common-context and traffic-related scenarios across COCO, Cityscapes, and Mapillary Vistas.Source: B. Qualitative Results · p. 11

COCO

• On COCO, Panoptic FCN preserves foreground and background details while handling objects at various scales.The results use pixel-by-pixel processing and a unified representation.Source: B. Qualitative Results · p. 11

Cityscapes

• On Cityscapes, Panoptic FCN depicts cars and pedestrians and performs satisfactorily on slender objects such as street lamps and traffic lights.Source: B. Qualitative Results · p. 11

Mapillary Vistas

• On Mapillary Vistas, Panoptic FCN performs especially well on vehicles and traffic signs in larger-scale traffic scenes.The qualitative results also exhibit instance-awareness and semantic-consistency.Source: B. Qualitative Results · p. 11