Source-linked AI summary
PMT: Plain Mask Transformer for Image and Video Segmentation with Frozen Vision Encoders
Niccolò Cavagnero, Narges Norouzi, Gijs Dubbelman, Daan de Geus
TL;DR
Encoder-only segmentation models are fast but require finetuning their encoders, undermining the reuse of frozen Vision Foundation Models across tasks. PMT addresses this with a lightweight Plain Mask Decoder operating on frozen features, matching frozen-encoder image accuracy and fully finetuned video accuracy while substantially reducing latency.
Problem
EoMT and VidEoMT require encoder finetuning, preventing a single Vision Foundation Model encoder from being reused across downstream tasks.
Method
PMT uses a lightweight Plain Mask Decoder to process learnable queries with frozen ViT features outside the encoder.
Results
PMT matches state-of-the-art frozen-encoder image segmentation while being up to 3× faster, and reaches video accuracy on par with fully finetuned methods at up to 8× the speed of frozen-encoder baselines.
Takeaways & Limitations
PMT preserves encoder-only simplicity and low latency while keeping the Vision Foundation Model encoder frozen and shareable across image and video segmentation.
Takeaways & Limitations
At the scale discussed, pre-trained encoders alone carry the representational capacity that makes specialized components largely redundant; smaller-scale settings are not established here.
Abstract
from arXiv · showhide
Vision Foundation Models (VFMs) pre-trained at scale enable a single frozen encoder to serve multiple downstream tasks simultaneously. Recent VFM-based encoder-only models for image and video segmentation, such as EoMT and VidEoMT, achieve competitive accuracy with remarkably low latency, yet they require finetuning the encoder, sacrificing the multi-task encoder sharing that makes VFMs practically attractive for large-scale deployment. To reconcile encoder-only simplicity and speed with frozen VFM features, we propose the Plain Mask Decoder (PMD), a fast Transformer-based segmentation decoder that operates on top of frozen VFM features. The resulting model, the Plain Mask Transformer (PMT), preserves the architectural simplicity and low latency of encoder-only designs while keeping the encoder representation unchanged and shareable. The design seamlessly applies to both image and video segmentation, inheriting the generality of the encoder-only framework. On standard image segmentation benchmarks, PMT matches the frozen-encoder state of the art while running up to ~3x faster. For video segmentation, it even performs on par with fully finetuned methods, while being up to 8x faster than state-of-the-art frozen-encoder models. Code: https://github.com/tue-mps/pmt.
1. Introduction
PMT addresses the incompatibility between encoder-only segmentation and frozen Vision Foundation Model encoders by moving query processing into a lightweight decoder. It preserves encoder-only simplicity and speed while matching strong segmentation baselines across image and video tasks.
- Motivation: EoMT and VidEoMT require full encoder finetuning because injected queries must be integrated into the encoder’s self-attention layers.Freezing the encoder prevents their query-processing mechanism from functioning effectively.
- Motivation: Finetuning task-specific encoders prevents one Vision Foundation Model from serving multiple downstream tasks or class definitions.Deployment otherwise requires separate finetuned encoders or inefficient task-specific decoders.
- Method: PMT introduces the Plain Mask Decoder, which processes learnable queries and frozen ViT features jointly through a small Transformer decoder.The decoder mimics the joint query-patch processing of the final encoder layers used by EoMT and VidEoMT.
- Results: Up to 3× faster, PMT matches state-of-the-art frozen-encoder accuracy for image segmentation.This result is reported across standard image segmentation benchmarks.
- Results: For video segmentation, PMT can compete with or outperform fully finetuned methods while remaining compatible with a frozen, shareable encoder.The frozen encoder supports multi-task deployment without changing its representation.
2. Related Work
Related work has progressively simplified image and video segmentation by replacing specialized components with query-based processing in ViT encoders. PMT extends this direction while addressing the requirement that those encoders remain frozen for reuse across tasks.
- Image Segmentation: Mask Transformer methods unify semantic, instance, and panoptic segmentation through learnable queries that predict class labels and binary masks.Queries alternate self-attention with cross-attention to image features before producing segmentation outputs.
- Image Segmentation: EoMT removes many task-specific image-segmentation components by injecting learnable queries into the final layers of a sufficiently capable ViT.It achieves competitive accuracy at significantly higher inference speed.
- Video Segmentation: VidEoMT extends the encoder-only approach to video by replacing specialized tracking modules with lightweight query propagation.It reports 5–10× speedups while preserving competitive accuracy.
- Video Segmentation: EoMT and VidEoMT still require full ViT finetuning, preventing their encoders from being reused for other downstream tasks.PMT addresses this limitation by moving query processing outside the encoder.
3. Method
PMT replaces encoder-injected queries with a standalone Transformer decoder operating on frozen ViT features, including features from multiple encoder depths. It supports image and video segmentation while preserving efficient query–patch processing and lightweight temporal propagation.
- Frozen-Encoder Incompatibility: Freezing the encoder makes encoder-injected query attention fundamentally incompatible because pretrained attention weights were not trained for the additional query tokens.The incompatibility is attributed to joint patch–query attention inside the frozen encoder.
- Plain Mask Decoder: PMT uses a standalone decoder that concatenates learnable queries with encoder patch tokens and processes them through standard multi-head self-attention.This jointly provides query self-attention, patch self-attention, and bidirectional query–patch attention without modifying the frozen encoder.
- Mask Prediction: During training, masked attention restricts queries to predicted regions, while mask annealing removes masking so inference uses a faster mask-free decoder.The decoder outputs query features and patch features used by the mask module for class and segmentation predictions.
- Lateral Connections: PMT extracts patch tokens from multiple encoder depths through lateral connections because fixed final-layer features may omit segmentation cues such as edges and boundaries.The extracted features are normalized, projected, and summed into a multi-depth representation for the decoder.
- Positional Encoding: Rotary positional embeddings are applied to decoder attention for spatial context, while query tokens remain position-free to preserve permutation invariance.Patch tokens retain encoder grid coordinates, and RoPE introduces no learnable parameters.
- Video Segmentation: PMT extends to online video segmentation by processing each frame with the frozen encoder and linking frames only through decoder query propagation.The temporal mechanism is inherited from VidEoMT, while lateral connections and decoding are applied independently per frame.
4. Experiments
Experiments show that PMT restores segmentation quality after frozen EoMT collapses, while retaining substantial speed advantages across image and video tasks. Its effectiveness depends on sufficiently large, extensively pretrained encoders, and shallow decoding is sufficient.
- Ablations: 6.8 PQ for frozen EoMT rises to 53.7 PQ after adding a standalone PMD, with only a small inference-speed reduction.Adding lateral connections further reaches 55.9 PQ at little computational cost.
- Pre-training and model size: 0.3 and 0.2 PQ separate PMT from frozen ViT-Adapter + Mask2Former with DINOv3 and DINOv2, respectively, while large-scale pretraining remains essential.The gap increases to 1.3 PQ with ImageNet-21K supervision and 2.4 PQ with ImageNet-1K supervision.
- Pre-training and model size: 0.3 PQ is the PMT–ViT-Adapter + Mask2Former gap with ViT-L, while PMT runs at approximately 3× the speed.The accuracy gap is 4.3 PQ with ViT-S and 0.8 PQ with ViT-B.
- Decoder depth: 54.0 to 55.6 PQ results from increasing the decoder from 2 to 4 layers, while 6 layers reach 56.1 PQ and 8 layers add no gain.The default decoder depth is therefore set to L_d = 6.
- Image benchmarks: 128 vs. 40 FPS gives PMT more than 3× the inference speed of the frozen ViT-Adapter + Mask2Former baseline on ADE20K semantic segmentation.PMT’s accuracy drops only 0.3 and 0.2 mIoU with DINOv2 and DINOv3, respectively.
- Video benchmarks: 65.7 vs. 64.9 mIoU establishes a new state-of-the-art result for PMT on VSPW video semantic segmentation with a frozen encoder.For video instance segmentation, PMT is also more than 8× faster than CAVIS and can outperform encoder-finetuned methods.
5. Conclusion
PMT reconciles encoder-only simplicity and speed with frozen VFM encoders by moving query processing into a lightweight decoder. It matches frozen-encoder image results and approaches finetuned video performance while enabling multi-task encoder reuse.
- Method: PMD replicates EoMT’s joint query-patch attention in a small module operating entirely on frozen features.PMT consequently leaves the ViT encoder untouched for downstream reuse.
- Results: PMT matches frozen-encoder image segmentation state of the art at up to approximately 3× higher inference speed.On video benchmarks, it achieves accuracy on par with fully finetuned methods at up to 8× the speed of frozen-encoder baselines.
- Scope: A sufficiently large and extensively pretrained encoder is required for the lightweight PMD to remain effective.This condition aligns with scaling observations reported for EoMT and VidEoMT.