A Generalist Framework for Panoptic Segmentation of Images and Videos

• Panoptic segmentation assigns each pixel a semantic class and an instance ID, combining semantic and instance segmentation into a scene representation.Source: §1 · p. 1
• Interchangeable instance IDs create a one-to-many mapping from an image to valid ID assignments, while traditional and end-to-end approaches use multi-stage pipelines or task-specific designs [34] [66] [40] [14].End-to-end approaches also rely on customized architectures and sophisticated losses [58] [17] [70] [16] [35] [68] [33].Source: §1 · p. 1
• The method generates discrete panoptic masks conditioned on an image, and adds past predictions as conditioning for streaming video, where it can learn to track and segment objects automatically.Source: §1 · p. 1
• A 512×1024 panoptic mask contains more than 1M discrete tokens, motivating a diffusion model that handles large masks directly using analog bits without an intermediate latent space.Source: §1 · pp. 1–2
• The proposed approach performs competitively with state-of-the-art methods in similar settings.Source: §1 · p. 2

• A panoptic mask has two channels: category labels and instance IDs, which are randomly reassigned during training because their permutations preserve the instances.Source: §2 · p. 2
• The model learns image-conditional panoptic-mask generation by maximizing the log probability of each training mask given its image.Masks may contain hundreds of thousands or millions of discrete tokens, making generative modeling challenging, particularly for autoregressive models.Source: §2 · p. 2
• Diffusion trains a denoising network and generates target data in parallel with fewer iterations than pixels, making it more effective for high-dimensional data than autoregressive generation.Source: §2 · p. 2
• Training samples noisy masks from clean data using noise, a uniformly sampled time, and a decreasing schedule, then learns a network to predict the clean data or noise with an ℓ2 loss.Source: §2 · p. 2
• At inference, generation starts from noise and repeatedly applies the denoising function through reverse state transitions to produce a sample.Source: §2 · p. 2
• Bit Diffusion converts discrete tokens into analog-bit representations for continuous diffusion, then quantizes generated values into categorical variables.Source: §2 · p. 2

3. Method

• Pix2Seq-D formulates panoptic segmentation as discrete data generation conditioned on pixels, adapting the Pix2Seq framing to dense prediction.The method is based on Bit Diffusion.Source: §3 · p. 2

3.1. Architecture

• Separating the image encoder from the mask decoder lets the encoder run once while the decoder iteratively refines the panoptic mask.The encoder maps pixels to feature vectors; the decoder reads out the mask from those features.Source: §3.1 · p. 2, Figure 2 · p. 3

3.2. Training Algorithm

• Training conditionally denoises analog bits representing noisy panoptic masks, and scaling them to b = 0.1 makes denoising harder than b = 1.0.The authors report that b = 0.1 works substantially better than the default scaling of 1.0.Source: §3.2 · p. 3
• The decoder is trained with softmax cross entropy over logits for base categories, while reverse diffusion uses its analog-bit predictions.This lets the network model the underlying category distribution rather than treating analog bits as real numbers.Source: §3.2 · p. 4
• Adaptive loss weighting gives tokens from smaller instances higher weight, with p = 0 yielding uniform weighting.The per-token weights depend on the instance pixel count and the tunable parameter p.Source: §3.2 · p. 4, Figure 4 · p. 4

3.3. Inference Algorithm

• Inference starts from random noise and iteratively refines the mask; only the decoder runs during those iterations.The time-interval parameter td is adjustable at inference time.Source: §3.3 · p. 4

3.4. Extension to Videos

• For streaming video, the model conditions on the current image and past mask predictions, learning to track and segment instances across frames without explicit instance matching through time.The past masks are concatenated with the noisy mask input, while the rest of the model remains unchanged.Source: §3.4 · p. 5, Figure 5 · p. 5

4.1. Setup and Implementation Details

• Experiments use MS-COCO, Cityscapes, and unsupervised DAVIS; models train mainly on COCO, transfer by fine-tuning, and use staged-resolution COCO training.Default sampling uses 20 steps on COCO and 32/8 on DAVIS.Source: §4.1 · p. 5

4.2. Main Results

• On MS-COCO, Pix2Seq-D is competitive with ResNet-50 specialists and significantly better and more efficient than UViM; on DAVIS, it is on par with state-of-the-art methods.The DAVIS comparison is for unsupervised video object segmentation.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6

4.3. Ablations on Training

• On MS-COCO, 0.1 scaling beats 1.0, cross-entropy substantially beats ℓ2, and p = 0.2 beats uniform loss weighting.The ablations compare input scaling, loss functions, and loss weighting.Source: §4.3 · p. 6, Table 3 · p. 6, Table 4 · p. 6

4.4. Ablations on Inference

• On MS-COCO, 20 inference steps are near-optimal; intermediate td values (2–3) perform best, while choosing a small-instance threshold brings only small gains.These are inference-time ablations.Source: §4.4 · p. 7, Figure 6 · p. 7
• On DAVIS, more steps help the first frame; using 32 first-frame and 8 later-frame steps reduces total sampling, while 8 then 1 still reaches J &F 67.3.The first frame requires more steps because of the cold start.Source: §4.4 · p. 7, Figure 7 · p. 7

4.5. Case study

• Examples on MS-COCO, Cityscapes, and DAVIS show Pix2Seq-D capturing small objects in dense scenes.Source: §4.5 · p. 7

5. Related Work

• Earlier image panoptic methods use multistage pipelines; end-to-end transformer methods still rely on customized mask-generation and fusion designs and matching losses.Source: §5 · p. 7
• Pix2Seq-D treats panoptic segmentation as image-conditioned discrete mask generation without task-specific inductive biases, yielding a simpler design extendable to video.Source: §5 · p. 7
• Video segmentation spans VOS, VIS, and VPS and combines segmentation with tracking; existing approaches use multistage pipelines or specialized end-to-end losses.Source: §5 · p. 7
• For UVOS, COCO-pretrained Pix2Seq-D is fine-tuned without manual initialization while architectures, losses, augmentations, and sampling remain largely unchanged.Source: §5 · p. 7
• Generalist vision models are mostly autoregressive, whereas Pix2Seq-D uses Bit Diffusion; earlier segmentation diffusion models did not model discrete masks or video segmentation.Source: §5 · pp. 7–8

6. Conclusion and Future Work

• The framework models image and video panoptic segmentation with conditional generative models of discrete masks.The experiments model 106 discrete tokens, a scale the paper says is difficult for other generative segmentation models.Source: §6 · p. 8
• The paper reports modeling 106 discrete tokens in its experiments, which it says is difficult for other existing generative segmentation models.Source: §6 · p. 8
• The empirical results remain behind well-tuned existing systems, although the approach is described as competitive and practically usable.Source: §6 · p. 8
• The authors identify the architecture, modeling choices, training procedure, and inference-step count as areas for further improvement.They mention progressive distillation as one technique for reducing inference steps.Source: §6 · p. 8

A. Algorithm details

• The appendix reproduces implementation details from Bit Diffusion for functions in Algorithms 1 and 2.Source: A. Algorithm details · p. 12
• Algorithm 3 is identified as the binary encoding and decoding algorithms.Source: A. Algorithm details · p. 12
• The `ddim_step` function estimates x at the next time using the current sample, a prediction, and the two time values.The step clips the prediction and applies a DDIM update.Source: A. Algorithm details · p. 12

B. More details on training and inference hyper-parameters

• MS-COCO mask-decoder pretraining uses 128×128 masks for 800 epochs, batch size 512, and scale jittering strength [1.0, 3.0].The stated shared settings include input scaling 0.1, loss weighting p = 0.2, learning rate 1e−4, and EMA decay 0.999.Source: B. More details on training and… · p. 12
• Cityscapes fine-tuning uses 800 epochs, batch size 16, and 1024×2048 images with 512×1024 masks, without scale jittering.Evaluation resizes predictions to 1024×2048 and filters annotations with fewer than 80 pixels.Source: B. More details on training and… · p. 12
• DAVIS fine-tuning uses 20k steps and batch size 32, with 512×1024 images and 256×512 masks.Evaluation runs inference 50 times and reports the mean; the stated standard deviation for J &F is around 1.5.Source: B. More details on training and… · p. 12
• Table 6 is labeled as Cityscapes validation-set results.Source: Table 6 · p. 12
• Table 7 is labeled as a Cityscapes validation-set ablation over image and panoptic-mask sizes, trained for 100 epochs.Source: Table 7 · p. 12

C. Results on Cityscape

• The Cityscapes validation comparison with prior work uses 1024×2048 images and 512×1024 masks for the main results.Source: C. Results on Cityscape · p. 12
• The resolution ablation finds that both larger image size and larger panoptic-mask size are important.Source: C. Results on Cityscape · p. 12

D. Results on KITTI-STEP

• KITTI-STEP video panoptic training retained DAVIS configurations at 384x1248; inference used td=1.5 and 10 sampling steps.These were the preliminary results reported in Table 8.Source: D. Results on KITTI-STEP · p. 12
• Pix2seq-D achieved decent KITTI-STEP results, though behind the state of the art, despite minimal tuning or changes.The comparison covers methods with ResNet-50 backbones.Source: D. Results on KITTI-STEP · pp. 12–13, Table 8 · p. 13
• Inference covered videos up to 400 frames; one longer validation video was split in two, which the authors expect to have negligible metric impact.Unlike methods that infer on video segments and stitch them afterward, Pix2seq-D processes the video in a streaming fashion.Source: D. Results on KITTI-STEP · p. 13

E. Extra Visualization

• Across two MS-COCO examples, iterative refinement makes panoptic outputs globally and locally consistent.Figure 11 shows predictions at steps 1, 2, 4, 8, and 16 of 20 total steps.Source: E. Extra Visualization · p. 13, Figure 11 · p. 14
• Additional prediction visualizations cover MS-COCO, Cityscapes, and DAVIS validation sets.Figures 12–13 show MS-COCO, Figure 14 Cityscapes, and Figure 15 DAVIS examples.Source: E. Extra Visualization · p. 13, Figure 12 · p. 14, Figure 13 · p. 15
• The examples label the image, prediction, and ground truth for visual comparison.These labels accompany the validation-set examples.Source: E. Extra Visualization · p. 15