Source-linked AI summary
VSCode: General Visual Salient and Camouflaged Object Detection with 2D Prompt Learning
Ziyang Luo, Nian Liu, Wangbo Zhao, Xuguang Yang, Dingwen Zhang, Deng-Ping Fan, Fahad Khan, Junwei Han
TL;DR
SOD and COD involve shared and distinct cues across multiple modalities, while prior work often relies on separate specialist models. VSCode jointly models these tasks with VST and 2D prompts that separate domain and task knowledge. It outperforms state-of-the-art methods across six tasks on 26 datasets and generalizes zero-shot to unseen tasks.
Problem
Existing SOD and COD research often uses task-specific specialist models despite shared information across related multimodal tasks.
Method
VSCode uses VST as a shared foundation model, 2D domain- and task-specific prompts, and a prompt discrimination loss for joint SOD and COD learning.
Results
VSCode surpasses existing state-of-the-art models across all tasks on 26 datasets and demonstrates zero-shot generalization to unseen tasks.
Takeaways & Limitations
A single generalist model can address multiple multimodal SOD and COD tasks while combining shared knowledge with task and domain peculiarities.
Takeaways & Limitations
Comparisons with EVP may not be completely fair because the methods use different training strategies and backbone utilization.
Abstract
from arXiv · showhide
Salient object detection (SOD) and camouflaged object detection (COD) are related yet distinct binary mapping tasks. These tasks involve multiple modalities, sharing commonalities and unique cues. Existing research often employs intricate task-specific specialist models, potentially leading to redundancy and suboptimal results. We introduce VSCode, a generalist model with novel 2D prompt learning, to jointly address four SOD tasks and three COD tasks. We utilize VST as the foundation model and introduce 2D prompts within the encoder-decoder architecture to learn domain and task-specific knowledge on two separate dimensions. A prompt discrimination loss helps disentangle peculiarities to benefit model optimization. VSCode outperforms state-of-the-art methods across six tasks on 26 datasets and exhibits zero-shot generalization to unseen tasks by combining 2D prompts, such as RGB-D COD. Source code has been available at https://github.com/Sssssuperior/VSCode.
1. Introduction
VSCode addresses the challenge of jointly modeling shared and task-specific knowledge across multimodal SOD and COD tasks. It uses VST with 2D prompts that separate domain and task peculiarities, supporting generalization across and beyond the trained tasks.
- Task Scope: SOD and COD span RGB, depth, thermal, and optical-flow domains, with each task represented by a domain dimension and a task dimension.The framework covers RGB SOD, RGB COD, RGB-D SOD, RGB-D COD, RGB-T SOD, VSOD, and VCOD.
- Motivation: Task-specific specialist models can overfit limited training distributions, motivating a generalist model that learns commonalities across tasks.Joint learning is presented as a way to use more data while reducing overfitting risk, although separating commonalities from peculiarities remains challenging.
- Approach: VSCode uses VST as a shared foundation model and 2D prompts to capture peculiarities separately along domain and task dimensions.The prompts are designed to combine for specific tasks and unseen tasks while the foundation model learns shared information.
- Approach: A prompt discrimination loss encourages 2D prompts to learn distinct peculiarities while allowing the foundation model to focus on commonality learning.This loss is introduced to reduce entanglement among prompts of the same type.
- Results: VSCode surpasses existing state-of-the-art models across all tasks on 26 datasets and demonstrates generalization to unseen tasks.The model is trained on four SOD tasks and two COD tasks, with an additional reserved task used for zero-shot evaluation.
2. Related Work
Prior SOD and COD research largely develops specialist models for individual modalities or tasks. VSCode instead jointly handles multiple tasks and modalities using 2D prompts that represent both domain and task differences.
- SOD and COD Methods: SOD research includes attention, multi-level fusion, recurrent, and multi-task methods across RGB, RGB-D, RGB-T, and video settings.RGB-D and RGB-T methods commonly incorporate complementary modalities through attention or feature fusion.
- SOD and COD Methods: COD research covers RGB, RGB-D, and video tasks using multi-task, multi-input, refinement, depth-inference, optical-flow, and specialized model designs.The related work distinguishes RGB COD, RGB-D COD, and VCOD as separate research settings.
- Prompt Learning: Prompt-based vision methods have mainly targeted individual tasks or modality-complementary settings, whereas VSCode assembles multiple multimodal SOD and COD tasks.The paper positions 2D prompts as a mechanism for zero-shot generalization to unseen tasks.
- Generalist Models: Generalist segmentation frameworks use queries, language or annotation prompts, or image-mask pairs, but primarily emphasize task differences.VSCode differs by separating both domain and task peculiarities while training all tasks simultaneously.
3. Methodology
VSCode extends VST into a unified multimodal SOD and COD model by combining shared features with domain- and task-specific prompts. Its encoder uses windowed prompt interactions, while task prompts also guide decoder reconstruction and are analyzed across depth.
- 3.1. Foundation Model: VSCode uses VST to learn shared features, 2D prompts to encode domain and task peculiarities, and a prompt discrimination loss to disentangle them.The architecture is designed for joint training across SOD and COD tasks.
- 3.1. Foundation Model: VST provides a transformer encoder, multimodal convertor, and transformer decoder, while VSCode adopts Swin, concatenates supplementary modalities, and projects them with an MLP.The decoder predicts saliency and boundary maps using VST’s token-based design.
- 3.2. Domain-specific Prompt: Domain-specific prompts for RGB, depth, thermal, and optical flow are appended to encoder window tokens and updated through regular or shifted window attention.Prompt tokens are replicated across windows before self-attention and the resulting features are passed to subsequent blocks.
- 3.3. Task-specific Prompt: Task-specific prompts preserve shared SOD and COD parameters while adding task-dependent encoder features and decoder attention for reconstructing object and boundary regions.Decoder prompts address the more intricate boundaries of camouflaged objects, and unlike saliency and boundary tokens they receive no direct supervision.
- 3.4. Prompts Layout and Discussion: Prompt correlations decrease with encoder depth for most domains and between SOD and COD, motivating deep prompts and progressively more tokens at higher layers.Lower layers share low-level cues, whereas higher layers learn more distinct semantic information.
4. Experiment
VSCode is evaluated across six training tasks and an unseen RGB-D COD task using benchmark comparisons, ablations, prompt-layout studies, and zero-shot testing. Results show that 2D prompts and prompt discrimination improve joint learning, while VSCode outperforms specialist and generalist methods across the six tasks and generalizes zero-shot to RGB-D COD.
- 4.3. Ablation Study: Domain-specific prompts substantially improve performance across datasets, while task-specific prompts significantly improve all COD tasks over special training.Task prompts produce smaller gains on some SOD tasks, and prompt discrimination improves most tasks.
- 4.3. Ablation Study: Removing encoder or decoder task prompts decreases performance, showing that both prompt locations contribute to the model.The 2D prompts add around 0.03M parameters, making them more parameter-efficient than the traditional special-training scheme.
- 4.3. Ablation Study: The best prompt layout uses shallow task prompts in the decoder and deep domain and task prompts in the encoder.Prompt-length experiments favor one domain token per block and a 1,1,5,10 encoder task-prompt layout for COD tasks; decoder performance declines beyond ten tokens.
- 4.4. Comparison with State-of-the-Art Methods: Comparing VSCode with EVP is not completely fair because the methods use different training strategies and backbone utilization.VSCode shares parameters across tasks, whereas EVP uses task-specific training.
- 4.4. Comparison with State-of-the-Art Methods: VSCode significantly outperforms specialist methods and two generalist models across all six evaluated tasks.The comparisons cover RGB SOD, RGB-D SOD, RGB-T SOD, VSOD, RGB COD, and VCOD benchmark suites.
- 4.5. Analysis of Generalization Ability: Zero-shot VSCode significantly outperforms the state-of-the-art specialist model on unseen RGB-D COD, while RGB-only inference performs considerably worse.The model combines depth and COD prompts despite RGB-D COD being excluded from training.
5. Conclusion
VSCode is a parameter-efficient generalist model that combines a foundation model with 2D prompts to learn shared and task-specific knowledge across multimodal SOD and COD. Experiments support its effectiveness on six training tasks and one unseen task.
- 5. Conclusion: VSCode combines a foundation model, 2D domain and task prompts, and prompt discrimination to address multimodal SOD and COD.The design aims to assimilate commonalities while disentangling task-specific knowledge.
- 5. Conclusion: Experiments demonstrate VSCode’s effectiveness on six training tasks and one unseen task.