Source-linked AI summary
Explicit Visual Prompting for Low-Level Structure Segmentations
Weihuang Liu, Xi Shen, Chi-Man Pun, Xiaodong Cun
TL;DR
Low-level structures such as manipulated, blurred, shadowed, and concealed regions are commonly handled with separate task-specific solutions. The paper proposes Explicit Visual Prompting, which adapts a frozen vision transformer using image-specific patch-embedding and high-frequency features, and reports strong performance across these tasks against efficient-tuning and task-specific alternatives.
Problem
Low-level structure segmentation tasks are typically addressed with separate domain-specific solutions, despite sharing useful low-level image clues.
Method
EVP freezes a pretrained vision-transformer backbone and prompts task adaptation with frozen patch-embedding features and each image’s high-frequency components.
Results
EVP achieves state-of-the-art performance across four low-level structure segmentation tasks and outperforms other parameter-efficient tuning methods.
Takeaways & Limitations
A unified frozen-backbone model can simplify low-level structure segmentation while remaining competitive with full fine-tuning and task-specific methods.
Abstract
from arXiv · showhide
We consider the generic problem of detecting low-level structures in images, which includes segmenting the manipulated parts, identifying out-of-focus pixels, separating shadow regions, and detecting concealed objects. Whereas each such topic has been typically addressed with a domain-specific solution, we show that a unified approach performs well across all of them. We take inspiration from the widely-used pre-training and then prompt tuning protocols in NLP and propose a new visual prompting model, named Explicit Visual Prompting (EVP). Different from the previous visual prompting which is typically a dataset-level implicit embedding, our key insight is to enforce the tunable parameters focusing on the explicit visual content from each individual image, i.e., the features from frozen patch embeddings and the input's high-frequency components. The proposed EVP significantly outperforms other parameter-efficient tuning protocols under the same amount of tunable parameters (5.7% extra trainable parameters of each task). EVP also achieves state-of-the-art performances on diverse low-level structure segmentation tasks compared to task-specific solutions. Our code is available at: https://github.com/NiFangBaAGe/Explicit-Visual-Prompt.
1. Introduction
Low-level structure segmentation spans forgery, defocus blur, shadow, and camouflage, but these tasks are usually handled with separate domain-specific solutions. EVP unifies them by prompting a frozen pretrained transformer with image-specific embedded and high-frequency features.
- Four low-level structure tasks are addressed jointly: forgery, defocus blur, shadow, and camouflaged-object detection.
- Prompting adapts a frozen foundation model to downstream tasks with few extra trainable parameters and can reduce model-storage requirements.
- The unified approach is motivated by shared low-level clues, including noise, compression, resampling, blur, shadow, and camouflage cues.
- EVP uses a frozen pretrained backbone and tunes features from each image’s patch embedding and high-frequency components.The design targets explicit image content rather than only dataset-level prompting.
- Experiments cover nine datasets across four tasks, with competitive performance against full fine-tuning and task-specific solutions.
- The method is reported to simplify low-level segmentation models while achieving comparable performance to well-designed state-of-the-art methods.
2. Related Work
Prior work addresses low-level structure segmentation through task-specific architectures and hand-crafted or learned cues. The reviewed tasks include forgery, defocus blur, shadow, and camouflaged-object detection.
- Visual Prompting Tuning: Visual prompting research largely studies learnable embeddings for vision recognition, whereas this work explores visual content for low-level structure segmentation.
- Forgery Detection: Forgery detection uses inconsistencies in noise, JPEG compression, SIFT, and resampling artifacts to identify manipulated pixels.
- Defocus Blur Detection: Defocus blur detection separates in-focus from out-of-focus regions using gradient or edge features, CNNs, and increasingly integrated deep-feature designs.
- Shadow Detection: Shadow detection research uses illumination and hand-crafted cues alongside CNN architectures and attention modules, with some methods adding lighting priors.
- Camouflaged Object Detection: Camouflaged-object detection is difficult because foreground and background patterns are visually similar, motivating texture, brightness, color, frequency, and deep-learning approaches.
3. Method
EVP adapts a frozen SegFormer backbone for low-level structure segmentation by tuning explicit features from frozen patch embeddings and image high-frequency components. Its architecture extracts and transforms these features, then uses Adaptors to produce prompts for transformer layers.
- 3. Method: EVP adapts a frozen SegFormer backbone using a small number of tunable parameters learned from frozen image embeddings and high-frequency components.The method is presented as adaptable beyond SegFormer to architectures such as ViT and Swin.
- 3.1. Preliminaries: High-frequency components are extracted by Fourier-transforming an image, masking frequency coefficients according to ratio τ, and applying the inverse transform.The image is decomposed into low- and high-frequency components, with the process applied independently to each RGB channel.
- 3.2. Explicit Visual Prompting: The patch embedding tune freezes the pretrained projection and adds a tunable linear layer Lpe that maps each original embedding to a c-dimensional feature Fpe.A scale factor r controls the number of tunable parameters.
- 3.2. Explicit Visual Prompting: The HFC tune learns a linear layer Lhfc to project high-frequency patches into c-dimensional features Fhfc.This module provides the high-frequency input used by the Adaptors.
- 3.2. Explicit Visual Prompting: Each Adaptor takes Fpe and Fhfc as inputs and generates prompting that attaches to a transformer layer.MLPi tune produces layer-specific prompts, while the shared MLPup matches transformer-feature dimensions.
4. Experiment
EVP is evaluated across four low-level structure segmentation tasks and compared with task-specific and efficient tuning methods. Ablations examine architecture components, tuning stages, parameter scaling, and backbone generality.
- Ablation Study: The complete prompting strategy combining the decoder, Fhfc, Fpe, and adaptor performs more effectively than architectural variants.Removing Fpe or Fhfc reduces performance, while sharing MLPi tuning saves parameters but causes a significant performance drop.
- Ablation Study: Performance improves as more SegFormer stages receive tunable prompting, with the largest improvement occurring from Stage1,2 to Stage1,2,3.The reported effect is positively correlated with the number of prompted transformer blocks.
- Ablation Study: r = 4 provides a balance between performance and model size.Performance improves as r decreases from 64 to 4, but further decreases to 2 or 1 do not consistently improve results despite larger models.
- Main Results: EVP outperforms full-tuning, decoder-only tuning, VPT, and AdaptFormer under comparable parameter budgets.With more parameters at r=4, EVP further outperforms full-tuning on 3 of 4 datasets.
- Backbone Generality: EVP also outperforms other tuning methods when SETR uses a plain ViT backbone.This evaluation uses ViT-Base pretrained on ImageNet-21k and compares EVP with VPT, AdaptFormer, and full-tuning.
5. Conclusion
The paper presents EVP as a unified solution for low-level structure segmentation. It uses explicit image-derived visual prompts with a frozen vision transformer and limited tunable parameters.
- Conclusion: EVP uses frozen patch-embedding features and high-frequency components from each input image as visual prompts.The method unifies forgery, defocus blur, shadow, and camouflaged object segmentation.
- Conclusion: A frozen ImageNet-pretrained vision transformer with limited tunable parameters achieves performance similar to full-tuned networks and state-of-the-art performance against task-specific methods.
A. Implementation Details
The implementation compares tuning variants built on a pretrained backbone and decoder, including full tuning, decoder-only tuning, VPT, AdaptFormer, and EVP. Evaluation uses standard segmentation metrics and standard-dataset training data, while EVP adds explicit prompting.
- Basic Setting: The basic model uses an ImageNet-pretrained feature-extraction backbone and a randomly initialized segmentation decoder.
- Tuning Variants: Full-tuning updates all encoder and decoder parameters, whereas Only Decoder updates decoder parameters alone.
- Efficient Tuning Methods: VPT optimizes dataset-shared prompt embeddings and the decoder, while AdaptFormer optimizes newly added AdaptMLP modules and the decoder.
- EVP: EVP adds explicit prompting to the pretrained backbone-decoder setup as described in the main paper.
- Metrics: The study reports AUC, F1, weighted F-measure, MAE, and E-measure, with AUC based on ROC true- and false-positive rates.
- Training Data: The method uses only standard-dataset training data and reports state-of-the-art performance despite several comparison methods using extra training data.
B. More Results
Additional experiments test explicit image-derived prompts beyond the main configuration. Gaussian filtering helps defocus blur, while a noise filter boosts performance in forgery detection.
- Explicit Prompt Variants: The experiments compare Gaussian filtering, a noise filter, an all-zero image, and the original image as explicit image prompts.
- Explicit Prompt Variants: Gaussian filtering performs better for defocus blur because it is also a kind of blur.
- Explicit Prompt Variants: The noise filter from forgery detection also boosts performance.
B.2. HFC v.s. LFC
The ablation compares high- and low-frequency prompting components. High-frequency components provide more general gains across several low-level structure tasks, whereas low-frequency prompting yields only trivial improvements.
- HFC v.s. LFC: Low-frequency components produce only trivial improvements on the evaluated datasets.
- HFC v.s. LFC: High-frequency components perform much better in shadow detection, forgery detection, and camouflaged object detection.
- HFC v.s. LFC: The comparison relates LFC to blur, paralleling the Gaussian-filter prompting result discussed in the accompanying ablation.
B.3. Mask Ratio τ
The mask-ratio ablation identifies 25% central-spectrum masking as the consistently strongest setting across tasks. Increasing the ratio can reduce performance, particularly for several detection tasks.
- Mask Ratio τ: 25% central-pixel masking in the spectrum achieves consistently better performance across all tasks.
- Mask Ratio τ: Performance may drop as the mask ratio increases, especially in shadow detection, forgery detection, and camouflaged object detection.
- Mask Ratio τ: The ablation compares HFC, LFC, and mask ratio τ, with FFT-based HFC consistently outperforming LFC.
C. Additional Visual Results
Supplementary visual results compare EVP with task-specific methods across forgery, shadow, defocus blur, and camouflaged object detection datasets.
- Additional visual results evaluate forgery detection on CAISA against ManTraNet and SPAN.
- Additional visual results evaluate shadow detection on ISTD against DSD, MTMT, and FDRNet.
- Additional visual results evaluate defocus blur detection on CUHK against BTBNet, CENet, and EFENet.
- Additional visual results evaluate camouflaged object detection on CAMO against SINet, PFNEt, JCOD, and RankNet.