PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Instance Perception as Object Discovery and Retrieval

Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, Huchuan Lu

arXiv:2303.06674v2cs.CV

TL;DR

Instance perception is fragmented across tasks with different prompts, datasets, and label vocabularies, limiting shared representations and efficient joint modeling. UNINEXT unifies these tasks through prompt-guided object discovery and retrieval, and reports superior performance across 20 benchmarks from ten tasks with one shared-parameter model.

  • Problem

    Task-specific instance perception methods fragment the field, hindering shared generic knowledge, cross-task collaboration, and joint training across diverse label vocabularies.

  • Method

    UNINEXT organizes ten tasks by prompt type and uses a unified prompt-guided object discovery and retrieval formulation with shared model parameters.

  • Results

    UNINEXT achieves superior performance on 20 challenging benchmarks from ten instance perception tasks using a single model with the same model parameters.

  • Takeaways & Limitations

    The unified formulation supports joint training across tasks and domains while avoiding task-specific heads and redundant parameters.

Abstract

from arXiv · show

All instance perception tasks aim at finding certain objects specified by some queries such as category names, language expressions, and target annotations, but this complete field has been split into multiple independent subtasks. In this work, we present a universal instance perception model of the next generation, termed UNINEXT. UNINEXT reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm and can flexibly perceive different types of objects by simply changing the input prompts. This unified formulation brings the following benefits: (1) enormous data from different tasks and label vocabularies can be exploited for jointly training general instance-level representations, which is especially beneficial for tasks lacking in training data. (2) the unified model is parameter-efficient and can save redundant computation when handling multiple tasks simultaneously. UNINEXT shows superior performance on 20 challenging benchmarks from 10 instance-level tasks including classical image-level tasks (object detection and instance segmentation), vision-and-language tasks (referring expression comprehension and segmentation), and six video-level object tracking tasks. Code is available at https://github.com/MasterBin-IIAU/UNINEXT.

1. Introduction

Instance perception spans ten diverse tasks, but task-specific definitions and models fragment the field. UNINEXT unifies these tasks as prompt-guided object discovery and retrieval, achieving superior performance across 20 benchmarks with one parameter-sharing model.

  • Task scope: Instance perception includes image, video, language-guided, and annotation-guided tasks that find objects or trajectories specified by queries.The ten tasks use category names, language expressions, or first-frame target annotations as references.
  • Motivation: Task-specific definitions and datasets hinder generic knowledge sharing, cross-task collaboration, and joint training across differing label vocabularies.The fragmented design also introduces redundant parameters and limits dynamic category changes during inference.
  • UNINEXT: UNINEXT reorganizes ten tasks into category-, language-, and annotation-prompted groups and solves them with unified prompt-guided object discovery and retrieval.The model first discovers proposals under prompt guidance, then retrieves instances using prompt-instance matching scores.
  • UNINEXT: A single UNINEXT model with shared parameters trains across tasks and domains, learning generic representations without task-specific heads.The formulation supports joint training on heterogeneous data and label vocabularies.
  • Results: UNINEXT achieves superior performance on 20 challenging benchmarks from ten instance perception tasks using the same model parameters.This result is reported as a central contribution of the paper.

2. Related Work

Prior work addresses instance perception through task-specific methods, while unified vision research develops broader learning paradigms or architectures for related task groups. These approaches motivate, but do not fully cover, the unified scope targeted by UNINEXT.

  • Instance perception: Detection and instance segmentation find categorized objects in images, whereas REC, RES, and R-VOS retrieve language-referred targets in images or videos.SOT and VOS instead track targets specified by first-frame boxes or masks.
  • Instance perception: Existing R-VOS methods process whole videos offline, which hinders applications involving long or ongoing videos.The limitation is described for current state-of-the-art approaches in referring video object segmentation.
  • Unified vision models: Unified vision models seek to solve multiple vision or multimodal tasks with one model and are commonly divided into unified learning paradigms and unified architectures.The related-work taxonomy frames UNINEXT within broader efforts toward generalizable and flexible vision systems.
  • Unified learning paradigms: Unified learning paradigms cover many tasks or modalities through approaches such as multi-task self-training, continuous learning, and sequence-to-sequence modeling.Examples include MuST, INTERN, Unified-IO, and OFA.
  • Unified model architectures: Unified architectures typically target closely related tasks, including detection and segmentation, multiple segmentation formats, or phrase grounding.Examples include Mask R-CNN, Mask2Former, Pix2SeqV2, and GLIP.

3. Approach

UNINEXT converts diverse prompts into a common representation, fuses them with visual features, and discovers and retrieves instances through a flexible query-based architecture. Its training combines staged pretraining and joint image/video learning.

  • Task formulation: The ten tasks are grouped by prompt type: category names, language expressions, or first-frame reference annotations.Category prompts cover detection, segmentation, MOT, MOTS, and VIS; language prompts cover REC, RES, and R-VOS; annotations cover SOT and VOS.
  • Overall framework: UNINEXT uses prompt generation, image-prompt feature fusion, and object discovery and retrieval as its three main components.This formulation treats all tasks as finding objects specified by prompts.
  • Prompt generation: Language prompts are encoded into embeddings, while annotation-guided inputs use a cropped 256 × 256 template with a binary target-prior channel.The reference encoder produces hierarchical features that are merged into the prompt-embedding format.
  • Image-prompt feature fusion: A bidirectional cross-attention module retrieves information between visual and prompt features and adds the retrieved representations to the originals.The resulting early fusion makes visual features prompt-aware and enhances prompt embeddings with image context.
  • Object discovery and retrieval: The decoder produces object proposals and instance predictions, including boxes, masks, and trajectory-association embeddings for relevant video tasks.Prompt-instance matching then retrieves the objects that correspond to the input prompt.
  • Object discovery and retrieval: Matching scores are computed between instance features and prompt-derived weights, replacing fixed-size classifiers and enabling joint training across diverse label vocabularies.Category prompts use category embeddings, while expression and annotation prompts use pooled prompt embeddings.
  • Training: Training proceeds through general perception pretraining, image-level joint training, and video-level joint training.The first stage uses Objects365, with auxiliary BoxInst losses because that dataset lacks mask annotations.

4. Experiments

UNINEXT is evaluated across 20 benchmarks spanning 10 instance perception tasks, using one model with shared parameters. It generally outperforms task-specific and prior methods, while ablations clarify when prompt fusion and query design matter.

  • Experimental Setup: UNINEXT uses the same model parameters across benchmarks covering object detection, instance segmentation, REC, RES, SOT, VOS, MOT, MOTS, VIS, and R-VOS.The experiments compare the unified model with task-specific counterparts and prior methods across 20 datasets.
  • Image-Level Tasks: 2.7 box AP separates UNINEXT from DN-Deformable DETR on object detection, while ViT-Huge reaches 51.8 mask AP for instance segmentation.The reported gains include 3.3 and 1.9 box AP over Cascade Mask-RCNN and ViTDet-H, respectively, and 4.3 AP over QueryInst with ResNet-50.
  • Language-Guided Tasks: 10.85 average RES points separate UNINEXT from LAVT, and UNINEXT with ResNet-50 surpasses previous approaches on all REC and RES splits.Stronger ConvNeXt-Large and ViT-Huge backbones produce new state-of-the-art results on both language-guided tasks.
  • Single-Object Tracking: 5.3 AUC and 5.8 P separate UNINEXT from TransT on TNL-2K, while stronger backbones achieve the best SOT AUC on all four benchmarks.The four SOT benchmarks are LaSOT, LaSOT-ext, TrackingNet, and TNL-2K.
  • Video Tasks: UNINEXT achieves reported gains of 3.0 mMOTA and 2.7 mIDF1 on MOT, and 6.1 mMOTSA on MOTS, over Unicorn.For R-VOS, ViT-Huge reaches 70.1 J &F on Ref-Youtube-VOS and 72.5 on Ref-DAVIS17, while operating online.
  • Ablations: Removing early feature fusion drops VOS by 21.4 J &F and also reduces REC and R-VOS performance, whereas static queries outperform dynamic queries by 2.8 AP on VIS.The unified model also performs significantly better than five retrained task-specific models on five tasks.

5. Conclusions

UNINEXT unifies 10 instance perception tasks through prompt-guided object discovery and retrieval, achieving superior performance on 20 challenging benchmarks with one model and shared parameters.

  • UNINEXT unifies 10 instance perception tasks with a prompt-guided object discovery and retrieval paradigm.
  • Extensive experiments show superior performance on 20 challenging benchmarks using a single model with the same model parameters.

A. Appendix

The appendix provides additional details on training, loss functions, network architecture, analyses, and visualizations for UNINEXT.

  • The appendix presents details about the training process, loss functions, network architecture, analyses, and visualizations.

A.1. Training Process

UNINEXT is trained in three stages, progressing from Objects365 pretraining to joint image and referring-expression fine-tuning, followed by diverse video-task fine-tuning.

  • The training process consists of three stages using the StepLR scheduler, which reduces the learning rate by a factor of 10 after specified steps.
  • Stage one pretrains UNINEXT on Objects365 for about 340K iterations, or 12 epochs, with a learning-rate drop at epoch 11.
  • Stage two jointly fine-tunes UNINEXT on COCO and RefCOCO/g/+ for 12 epochs.
  • Stage three fine-tunes UNINEXT on diverse video-level tasks using equal sampling ratios for SOT&VOS, MOT&MOTS, VIS, and R-VOS.

A.2. Loss Functions

UNINEXT combines retrieval, box, and mask losses across training stages, adding embedding-based association for instances across video frames and box-only mask learning for Objects365.

  • Lretrieve and Lbox are used across all three training stages.
  • UNINEXT adds Lembed in the final stage, using contrastive loss to train embeddings that associate instances across frames.
  • Lretrieve converts raw instance-prompt matching scores into sigmoid probabilities and applies Focal loss.
  • Lbox combines GIoU Loss and ℓ1 loss, with GIoU using the smallest enclosing box and union areas.
  • For mask-annotated datasets, UNINEXT adopts Focal Loss and Dice Loss; for Objects365, it uses Projection Loss and Pairwise Affinity Loss from box-level annotations.
  • The appendix describes a mask-pair probability based on predicted masks, while ye = 1 indicates matching ground-truth labels and Se represents edge color similarity.

A.3. Network Architecture

UNINEXT uses an encoder-decoder Transformer with segmentation and matching enhancements to produce final instance predictions. Its design also incorporates IoU-based scoring and DINO techniques.

  • A.3. Network Architecture: UNINEXT transforms enhanced visual and prompt features into final instance predictions with an encoder-decoder Transformer.The architecture is based on two-stage Deformable DETR.
  • A.3. Network Architecture: A dynamic-convolution mask head generates instance masks from parameters predicted from instance embeddings.An MLP produces parameters used in three 1×1 convolution layers over feature maps.
  • A.3. Network Architecture: SimOTA replaces one-to-one Hungarian matching, allowing multiple queries per ground-truth object and providing more positive samples.NMS removes duplicate predictions during inference.
  • A.3. Network Architecture: An IoU branch estimates box quality, while inference scores combine instance-prompt matching and IoU scores through their geometric mean.IoU does not affect label assignment during training.
  • A.3. Network Architecture: UNINEXT adds DINO techniques including contrastive DN, mixed query selection, and look forward twice to further improve performance.

A.4. Analysis and Visualizations

UNINEXT achieves state-of-the-art performance across 10 instance-level tasks and supports retrieval by category names, language expressions, and target annotations. These prompts enable flexible perception, localization, tracking, and segmentation across varied scenarios.

  • Analysis: UNINEXT outperforms competitive counterparts and achieves state-of-the-art performance on all 10 instance-level perception tasks.The comparisons cover image, language, and video tracking task counterparts.
  • Retrieval by Category Names: Category-name prompts restrict perception to the requested object categories, such as dining tables, wine glasses, cakes, and knives.The formulation also has potential for zero-shot open-vocabulary detection, which the paper leaves for future work.
  • Retrieval by Language Expressions: Language-expression prompts locate referred targets among similar distractors while capturing positional and size relationships.Examples include relations such as left, middle, right, and taller.
  • Retrieval by Target Annotations: First-frame box or mask annotations enable UNINEXT to track and segment targets in complex video scenarios.Box annotations are converted into a binary target prior for tracking.
  • Retrieval by Target Annotations: The model supports target retrieval from both box annotations for SOT and mask annotations for VOS.
Loading 2303.06674v2…